{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 玉树芝兰+Python攻城狮\n",
    "玉树兰芝和python攻城狮两个公众号都是以学习和分享python知识，数据挖掘为主要内容，通过爬取他们的文章，提取他们的关键词，包括数据科学,Python,数据分析,可视化 科研入门知识,选题,分析文献，效率，笔记，写作，书籍，就业，爬虫等等。\n",
    "![玉树兰芝](https://gitee.com/Zkimzie/selenium_Official-Accounts/raw/master/img/%E7%8E%89%E6%A0%91.png)\n",
    "![python](https://gitee.com/Zkimzie/selenium_Official-Accounts/raw/master/img/python.png)\n",
    "通过关键词的数据，分析两个公众号喜欢分享什么关键词为主，除去无法分类的文章，玉树芝兰python文章有38篇，机器学习、数据科学分别有9篇和7篇，而python攻城狮python文章有47篇，数据分析有9篇，因而可根据我们的需求去关注公众号，对未来的学习会有所帮助。\n",
    "\n",
    "# 数据最小可用产品\n",
    "通过selenium和python的结合，成功获取到公众号玉树兰芝的41页，每页基本5篇文章以上，共抓取205条数据量，python攻城狮29页，共176条数据量，都包含文章标题、文章发布时间、文章链接，并对文章进行分类，通过此次小的测试，看看两个订阅号主要分享哪些领域的知识，以解决同学们能根据需要去关注公众号，解决相关问题。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "公众号 = \"玉树芝兰\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "fn = { \"output\" : { \"公众号_htm_snippets\": \"data_raw_src/公众号_htm_snippets_{公众号}.tsv\",\n",
    "                    \"公众号_df\": \"data_raw_src/公众号_df_{公众号}.tsv\",\n",
    "                    \"公众号_xlsx\": \"data_sets/公众号_url_{公众号}.xlsx\" } \\\n",
    "      }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 采集公众号（requests）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "{'app_msg_cnt': 203, 'app_msg_list': [{'aid': '2653540806_1', 'album_id': '1339195592689549313', 'appmsgid': 2653540806, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAvib7401VPHibcicChaa9Mow607g2JzoZ2aGCI9Pq3a46N6RNZnBzNa0k5tiaTj25eiapEVzKV71QjCkQQ/0?wx_fmt=jpeg', 'create_time': 1589360832, 'digest': '要让你的优秀作品在选拔中替你说话。而好的作品，需要时光的雕刻。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540806&idx=1&sn=2e40ec2d254c63d1c95856208814adfd&chksm=f389bb11c4fe3207d9269f229588ff7d592f5268616e9f91e4506c72655af8d0488738f78f1e#rd', 'tagid': [], 'title': '大一就开始准备考研，合适吗？', 'update_time': 1589360832}, {'aid': '2653540789_1', 'album_id': '1339188083526434816', 'appmsgid': 2653540789, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAtfds6VFs8mM2EI4tv51z43ySzmzOveFnwq9Pia3G1UOIsm1Zht1kZUCtzG9p9AicMWnAsia9qGFstmw/0?wx_fmt=jpeg', 'create_time': 1588674888, 'digest': '推荐给你一款最新的免费工具，解决你英文改写润色的后顾之忧。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540789&idx=1&sn=d2562da2dc5e86a69ae16a028c02915b&chksm=f389bb62c4fe32749c23fd6d70c8a4fcbd10e3969c5e6eb2b5f6e86bf8bc5d19ac7e374981e3#rd', 'tagid': [], 'title': '如何用人工智能免费帮你改英文？', 'update_time': 1588674888}, {'aid': '2653540778_1', 'album_id': '1339183650063220738', 'appmsgid': 2653540778, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAthWnZzHYEHEYo12EcrOmBHr3LFpsia0ybuVJXLsowiaYOWaJZ3zc8BeOQeAttZ31TSicaBlTiaNdABtQ/0?wx_fmt=jpeg', 'create_time': 1588072166, 'digest': '怎样把琐碎的「文献采集」、「标注笔记」、「整合引用」，和「样式输出」一气呵成地自动化搞定？', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 1, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540778&idx=1&sn=b614b519ac2e30ece6a020ee9a2cf716&chksm=f389bb7dc4fe326b179498bee21774e7f0b10b3c761e4661944c530fcfc2f7f378d2f947340d#rd', 'tagid': [], 'title': '如何高效做文献回顾？（工具篇）', 'update_time': 1588072166}, {'aid': '2653540768_1', 'album_id': '0', 'appmsgid': 2653540768, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAstPZ1OqLj8kx9CSiafv9GVNuD4yjK93JichSBIUcECBGMaJRiag3UrrXSZ12d3dHQ9F2yaNcy4m768w/0?wx_fmt=jpeg', 'create_time': 1587391737, 'digest': '「工欲善其事，必先利其器。」想要用好卡片式笔记进行学习与创作，没有一款趁手的工具怎么行？这一讲，我为你介绍卡片式笔记工具 Roam Research 的一些基本功能。希望你能感受到我第一次看到它时的惊艳。', 'has_red_packet_cover': 0, 'is_pay_subscribe': 0, 'item_show_type': 5, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540768&idx=1&sn=f4cdff85383be13c41662f0ab4f522b6&chksm=f389bb77c4fe32611d4666dbb30985bd01accdf354319721cbddc0ab50a46f72052f5c65ff5e#rd', 'tagid': [], 'title': '卡片式笔记工具 Roam Research 介绍', 'update_time': 1587391737}, {'aid': '2653540764_1', 'album_id': '1339188083526434816', 'appmsgid': 2653540764, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAvYf0RttvmH3uNia5Zbr3iahr4CycibA9ecma8vLvS6K9Ks5klB8zquMuveiaZfF5BT7qr0OxsjhKB4Lw/0?wx_fmt=jpeg', 'create_time': 1586647921, 'digest': '会当凌绝顶，一览众山小。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540764&idx=1&sn=b192edb5b54b2c04f814fa1b4e94bf02&chksm=f389bb4bc4fe325dbd969500dce4f9efff1217de584340a4a2a5e974078b08d377533b1f7aa4#rd', 'tagid': [], 'title': '如何交互可视化你的卡片式笔记网络？', 'update_time': 1586647921}], 'base_resp': {'err_msg': 'ok', 'ret': 0}}\n",
      "5\n",
      "{'app_msg_cnt': 203, 'app_msg_list': [{'aid': '2653540753_1', 'album_id': '0', 'appmsgid': 2653540753, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAsmNBmKdWVaZiaG5QVru1MY3b0C9XjPMvHsPy7GVVDohXCrSFYhqUDItiaDCSXgaHVgzvfVNUp8FDcw/0?wx_fmt=jpeg', 'create_time': 1585188990, 'digest': '你好！我是王树义老师。最近流行线上教学。有位朋友准备尝试把线下培训转移到线上，希望我能给提出一些建议。我于是做了这个分享。如果你正准备开始线上教学，希望这个视频能帮你少踩一些坑。如果你已经有了一段时间的教学经验，我也希望和你切磋交流，一起提升咱们的线上教学质量。如果你对本视频讲述的内容有疑问，或者要提出批评建议，欢迎留言。', 'has_red_packet_cover': 0, 'is_pay_subscribe': 0, 'item_show_type': 5, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540753&idx=1&sn=a0cdbe7c918e83e34b18881ebe50f272&chksm=f389bb46c4fe32508a6981d4fe0502905bd9a9f8900501bf5f9d454345f7ebcc7228e1a5b33f#rd', 'tagid': [], 'title': '线上教学心得', 'update_time': 1585188990}, {'aid': '2653540745_1', 'album_id': '1339195592689549313', 'appmsgid': 2653540745, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAvEcelEQLnnIuJCbeIpLpjO67Mf5yKeLFgt4bfHwwibMjOJFDtd64AmhMeg6H6z51WLBXB6tYJoia6g/0?wx_fmt=jpeg', 'create_time': 1584072711, 'digest': '我有一个坏消息，和一个好消息，你想先听哪个？', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540745&idx=1&sn=e0f4f6224834dce897bfb9a441334aef&chksm=f389bb5ec4fe32487faa3b35ac2bbe55b72327ac3a46c7dc41b65e97fc330f1d23d4e21d9ef8#rd', 'tagid': [], 'title': '扩招了，你的研究生学历会不会贬值？', 'update_time': 1584072711}, {'aid': '2653540740_1', 'album_id': '1339183650063220738', 'appmsgid': 2653540740, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAuUPZbgcjCYwib0v1I3BZKOXnLUiaPAAKBlAYXYBiaibzfMtXBKZxTSGI71ricRs0FpH1c26IUX4ibChErA/0?wx_fmt=jpeg', 'create_time': 1583760290, 'digest': '横看成岭侧成峰，远近高低各不同。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540740&idx=1&sn=67869ae1f36c45f06c637e07fe775a2f&chksm=f389bb53c4fe32451762eafdd83c7699245dc73d60f95080f3fe0d6220f511e9ad1ee0fe5bf6#rd', 'tagid': [], 'title': '如何用卡片法写论文？', 'update_time': 1583760290}, {'aid': '2653540733_1', 'album_id': '1339188083526434816', 'appmsgid': 2653540733, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAuiaE6hDlry6ic21BvLaxgbF131bzPyUEs0fSNYsdqDl7TO279qhJVDKFMoVG0ov3vN8BqicaW9qdl7w/0?wx_fmt=jpeg', 'create_time': 1582636402, 'digest': '这种诸多写作高手推荐的方法，实践起来为何这么困难？咱们来聊聊。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540733&idx=1&sn=2d3ccf04a0086c6137b06c731e0805c4&chksm=f389bbaac4fe32bc2c7e565dbfe954474dc2f16d884b0f6743f2adfe3447b0109fd74901b941#rd', 'tagid': [], 'title': '如何高效实践卡片式写作？', 'update_time': 1582636402}, {'aid': '2653540728_1', 'album_id': '0', 'appmsgid': 2653540728, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAuY2VrwcyA28icUeGicYLMkicbjhUI4VdcloicMTNBDHOB5nqeyp5dGrNibjt996qagib5GjoJ5nXuE8UcQ/0?wx_fmt=jpeg', 'create_time': 1581322334, 'digest': '希望这些推荐，可以帮助你更加安心、健康地宅在家里。为抗击疫情、阻断病毒、安定社会，做出自己的一份贡献。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540728&idx=1&sn=339f2e2406426c5e79ad855a6551ad26&chksm=f389bbafc4fe32b9f5982782aec37ed36757f8d3bca9c773576f6ae29cf225c6a77fbcff63cd#rd', 'tagid': [], 'title': '抗疫宅家，给你推荐几个隔离工具', 'update_time': 1581322334}], 'base_resp': {'err_msg': 'ok', 'ret': 0}}\n",
      "10\n",
      "{'app_msg_cnt': 203, 'app_msg_list': [{'aid': '2653540724_1', 'album_id': '0', 'appmsgid': 2653540724, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAtOdOLWAm4zzNeD9AmV9Lche5Tw5AlTQDAPotejd1o0iaJuAvyVnsNKWNRb1yzC6LlmaDaAL4yMDxQ/0?wx_fmt=jpeg', 'create_time': 1581150496, 'digest': '推荐给你一门好课。帮你在研究方法武器库中，添上复杂系统仿真这一项。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540724&idx=1&sn=713050c012d7724bf8a3240991b05859&chksm=f389bba3c4fe32b54f27599185bfd09d31025a86f78f585c10c95a6ada96d663a8a44e58b3ed#rd', 'tagid': [], 'title': '如何高效入门复杂系统仿真？', 'update_time': 1581150496}, {'aid': '2653540719_1', 'album_id': '1339178811027587075', 'appmsgid': 2653540719, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAtjvnxO9nPT8iaAeibiaG6B4iaWgfoNdT2ZDoSdrkzqibDrxGActYVlG11LbLpiaLdpm13XhsmjbXSqXD7Q/0?wx_fmt=jpeg', 'create_time': 1579502752, 'digest': '不用学前端编程，你就能用 Python 简单高效写出漂亮的交互式 Web 应用，将你的数据分析成果立即展示给团队和客户。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540719&idx=1&sn=226b8ffaa92d7b21a81d3d3bda28ba6b&chksm=f389bbb8c4fe32ae1c192fa5c2888b54057b6ecb02bf63c20ea5930fa204c792bf33b24f0fbd#rd', 'tagid': [], 'title': '如何用纯 Python 写 Web 应用？', 'update_time': 1579502752}, {'aid': '2653540714_1', 'album_id': '1339188083526434816', 'appmsgid': 2653540714, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAtdgrJqYOuIMA2ogE2OUrhYrKibN7tBFtzGRM0vCDHHEFe9thLHkiaQ4beRxHtNwF8ZmicKPmptxWBMA/0?wx_fmt=jpeg', 'create_time': 1579265586, 'digest': '想把 iPhone 上的操作流程录制下来，通过朋友圈分享给小伙伴？这几个小技巧可能对你有用。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540714&idx=1&sn=bfa3615317cfaf371386a52312d4f059&chksm=f389bbbdc4fe32ab00d97bd1b34bfad67115afa995d76797818a359bfa4c2aaa193312c2fd5f#rd', 'tagid': [], 'title': '如何在朋友圈演示你的 iPhone 操作过程？', 'update_time': 1579265586}, {'aid': '2653540707_1', 'album_id': '1339188083526434816', 'appmsgid': 2653540707, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAsDgZVNXkeEcIXHCDqR8MUSqCj5UnTvVWuicHE18NyLovYJia5KowoR0hqL7O4QJYCcHuBfyZM19jlg/0?wx_fmt=jpeg', 'create_time': 1577197630, 'digest': '数据分析的门槛在降低，希望你能及时意识到。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540707&idx=1&sn=5253093da915156d28c9f5bb91d957e1&chksm=f389bbb4c4fe32a25b71b802613b972ae9514ac935092630820fc985f56e782074331833f9dc#rd', 'tagid': [], 'title': '如何不写 SQL ，探索和分析数据库？', 'update_time': 1577226000}, {'aid': '2653540699_1', 'album_id': '0', 'appmsgid': 2653540699, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAvYGfv1DQGBO3JIeLWGNj908uJQHwUqmZw1GbuicsmoyhHkgkGOs5ySoptjwP9x1SI324W79APia6Ig/0?wx_fmt=jpeg', 'create_time': 1576208645, 'digest': '祝好运气！', 'has_red_packet_cover': 0, 'is_original': 0, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540699&idx=1&sn=b4cc0ea725350b93d14ee3eb0b5d9180&chksm=f389bb8cc4fe329ac6289b9fa74827278d0f84fbc81e67e655661e8762cdc515af23eb0b39d9#rd', 'tagid': [], 'title': '【福利】《巴拉巴西成功定律》抽奖赠书', 'update_time': 1576208645}], 'base_resp': {'err_msg': 'ok', 'ret': 0}}\n",
      "15\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'app_msg_cnt': 203, 'app_msg_list': [{'aid': '2653540692_1', 'album_id': '0', 'appmsgid': 2653540692, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAvrnl9s5GfM5HsDXFibP1KmjcjEic8HFPnxlca3gAawQCTs7VBZwTOicNzw5ic2Dt14TESMfibB8ickXusA/0?wx_fmt=jpeg', 'create_time': 1576159121, 'digest': '给你推荐一本“武器级”图书。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540692&idx=1&sn=21c296472df734f2d82f30b9e9af04d8&chksm=f389bb83c4fe3295bfed33191fe4f48e673034a2dc86b9b2d4082eebc24a69558f691cc678ef#rd', 'tagid': [], 'title': '如何科学地获得成功？', 'update_time': 1576159121}, {'aid': '2653540688_1', 'album_id': '0', 'appmsgid': 2653540688, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAtyI6hsg2YdqaAsZAZ7dEX846jf0a9r4yoljLh21L88Mia81E04zXcfbjz7NkQ6nDKPO1viah8X571Q/0?wx_fmt=jpeg', 'create_time': 1575895803, 'digest': 'PyTorch 入门，坑着实不少。咱们来谈谈，如何选个合适的教程，避开它们。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540688&idx=1&sn=668e9a979b79dc589c6e5b74aff92bcd&chksm=f389bb87c4fe3291d4ff1f1e34178362e91273e2ce3849da546ae6743039877acef6a8da50dc#rd', 'tagid': [], 'title': '如何高效入门 PyTorch ？', 'update_time': 1575895803}, {'aid': '2653540684_1', 'album_id': '1339188083526434816', 'appmsgid': 2653540684, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAs95Eib8kupxoa1Q3vnvWuCsV45tBDeVbTnibugUPaockJZgLgyhGpiaQAA1QboSkaHVbbIswmCWes1g/0?wx_fmt=jpeg', 'create_time': 1574072422, 'digest': '毕业论文字数要求太多，写起来很苦恼？把我新发现的写作利器分享给你。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540684&idx=1&sn=36a64c276264680fa54f060970118f94&chksm=f389bb9bc4fe328d4bc245718e086a40b9ba27ace93e967bc9de281d06802264f88f8a8b60dd#rd', 'tagid': [], 'title': '如何高效写长文？', 'update_time': 1574072422}, {'aid': '2653540680_1', 'album_id': '1339183650063220738', 'appmsgid': 2653540680, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAsN7r4UzicwNR7Np7P7UD8iamghsDBvQlkTDpdl0eTOSbYqYVlBhbibBBMSmwA7AU77Bhtv2Jraq21mg/0?wx_fmt=jpeg', 'create_time': 1572915307, 'digest': '悄悄告诉你几个窍门。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540680&idx=1&sn=6dfb8f84507c5c974f4a3eb8300ba22c&chksm=f389bb9fc4fe328931fe1193d4819d2cda36e9af82f0aac39a9913b8705ec7df439c944fadd3#rd', 'tagid': [], 'title': '论文读不懂怎么办？', 'update_time': 1572915307}, {'aid': '2653540676_1', 'album_id': '1339183650063220738', 'appmsgid': 2653540676, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAscVvTI2lJdDAgzWtaBmgeBYHqd7NoJctbqMcwtC5WDNE07gibMpsmH4r8kHktymEXCFQmgt6WkL6w/0?wx_fmt=jpeg', 'create_time': 1571644554, 'digest': '用产品经理的视角，找寻论文专业读者的知识需求。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540676&idx=1&sn=7ff14b24e2a1987141385b2dd4e6a738&chksm=f389bb93c4fe32859369249715b6104144bea7a364122cc58a0daad9504c79d34e74c993ddf8#rd', 'tagid': [], 'title': '如何与导师有效沟通你的论文选题？', 'update_time': 1571644554}], 'base_resp': {'err_msg': 'ok', 'ret': 0}}\n",
      "20\n",
      "{'app_msg_cnt': 203, 'app_msg_list': [{'aid': '2653540672_1', 'album_id': '1339188083526434816', 'appmsgid': 2653540672, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAtD2h49twibLXhsYVDZPLDCicAqn9ib9LpG0bBzUzY29VuYlVJ6TT1Z220A6muiay31rmE61K3iaiakRQqA/0?wx_fmt=jpeg', 'create_time': 1570284102, 'digest': '工欲善其事，必先利其器。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540672&idx=1&sn=5be1b09e9ea2e07d1fa9eec72d83e806&chksm=f389bb97c4fe3281c2d3826ab197dc5780bfac5651815e30b67efdd9aa3417fde71c05868b10#rd', 'tagid': [], 'title': '如何高效写作公众号文章？', 'update_time': 1570284101}, {'aid': '2653540664_1', 'album_id': '1339178811027587075', 'appmsgid': 2653540664, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAsN0LR2icSqBE9DykHibtCCiaRbwXTMXGd8d4cjmcNa4B8TOnPqM5XDpqVH9URdObxIrD5Wpqut9azCw/0?wx_fmt=jpeg', 'create_time': 1569423942, 'digest': '简单、直接、高效地从文本中提炼关键内容。让你体验正则表达式的威力。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540664&idx=1&sn=fbd0088427d8133b80964e010df2494d&chksm=f389bbefc4fe32f9a2b1255779e67cbfb49259a6d43e1261d94abcc0ae4ea8adef87719038cd#rd', 'tagid': [], 'title': '如何用 Python 和正则表达式抽取文本结构化信息？', 'update_time': 1569450600}, {'aid': '2653540660_1', 'album_id': '1339183650063220738', 'appmsgid': 2653540660, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAvQZ7b6ejaQbvcluGp4oib1MHejsELabnqXs90DMgnC1loWt2bT0KDZsqFg6vItJFzptKoxYDgPhicg/0?wx_fmt=jpeg', 'create_time': 1568599343, 'digest': '四大心法，助你写出更好的“第一封邮件”。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540660&idx=1&sn=fef006a155a52385c124394b5187d48d&chksm=f389bbe3c4fe32f5c91ab7c4fcbe52a5a02a6cf4b17130626f9163253302de16ee453c08be57#rd', 'tagid': [], 'title': '给导师的套磁信该怎么写？', 'update_time': 1568599343}, {'aid': '2653540656_1', 'album_id': '0', 'appmsgid': 2653540656, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAt3jsNzUCrtKsF92icYqOOQkvVYePBhKftLceNjNXFpqVpjibXmnwH9zPkPXDqahHBnfrQEZc9j42IQ/0?wx_fmt=jpeg', 'create_time': 1567952134, 'digest': '解决眼高手低，促进触类旁通。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540656&idx=1&sn=336e3c9284b15cf24c38d063468ed79e&chksm=f389bbe7c4fe32f1df04ae208ab4350d95503db8deeff3e8c44b3be164af53823f47067e2d6c#rd', 'tagid': [], 'title': '“没有灵魂”的AI作诗，有啥用？', 'update_time': 1567952133}, {'aid': '2653540651_1', 'album_id': '0', 'appmsgid': 2653540651, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAtGjgSicRCRsty9hpY7eQhgGTJMibOcK6d4GkjRUOw26sQhZ7Emzic0rvA75wN2lC8wga9zAl7HmYrAw/0?wx_fmt=jpeg', 'create_time': 1566999097, 'digest': '听好了，你只有两个选项……', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540651&idx=1&sn=5f729b88363d17ca199e10906756331a&chksm=f389bbfcc4fe32eac8cbb14e7b4df586914cc823fa10a160f4c19edc0c8ad450cf3af5be738c#rd', 'tagid': [], 'title': '美国人喜欢工会吗？', 'update_time': 1566999097}], 'base_resp': {'err_msg': 'ok', 'ret': 0}}\n"
     ]
    }
   ],
   "source": [
    "# 目标url\n",
    "\n",
    "import time\n",
    "import requests\n",
    "import pandas as pd\n",
    "import csv\n",
    "\n",
    "\n",
    "url = \"https://mp.weixin.qq.com/cgi-bin/appmsg\"\n",
    "\n",
    "# 使用Cookie，跳过登陆操作\n",
    "#更换cookie、header、token（每一次都会不同）\n",
    "headers = {\n",
    "  \"Cookie\": \"pgv_pvi=3297818624; pgv_pvid=7709096940; tvfe_boss_uuid=48cc16cb3ec1f962; ua_id=tlTvVwoNtM20N1AjAAAAADDBb7pu824i7Vficjn6qBE=; mm_lang=zh_CN; wxuin=71730836719479; RK=mMipTQ5sHY; ptcz=70879bb122a7b70b97f77b1b72255caa4ad6156b86eaafd0ec887e1de931cb58; pgv_si=s6941804544; vfwebqq=bdda92886d0cfb545d666481121c22539d4a9e270127c25d032bc6c95d821cc01d5c33b165944942; rewardsn=; wxtokenkey=777; pgv_info=ssid=s7658580900; qqmusic_fromtag=66; uin=o0515695070; skey=@7E8Qwj2DT; cert=8NSLCqkGK246wYVsOdFRwQTC5UyKhMAk; ticket_id=gh_d7be5b31e2e1; noticeLoginFlag=1; uuid=dd15770cf6ddc43cc1718f24572be3cf; bizuin=3887083431; ticket=8322f9bc3144a37ae14e26d23fc03e9ddc031a72; rand_info=CAESIHi5PryMFA8vL4CabU3n3y/82zPuStGfdZBvSf6w5I4Q; slave_bizuin=3887083431; data_bizuin=3887083431; data_ticket=c+WXv5TKmGvHM4IVJsuLxLcksmEELH8BcjoWcHYb9lldQpqiZePUNnpyngd/LG2Z; slave_sid=dXkybU1kR200MWdyNW1IRHF6TFM2M0xxU2djdEM5SkI4YVJZMmpEbGNKSENVczRLV2k4T01ONUpvYloxUWtRSmJiUUJUTHZhZ2EyS09tMkpSbF9WNkNWVDBGSEZ6Zm5sb3BpN0JPS1BHeklfbFlvbHVXZEZ1VjFpUGJ2NXBGdjJONHpNT21jQnFUVzJPOW1B; slave_user=gh_d7be5b31e2e1; xid=7f5c784aecd239ec6df6663008f02bde; openid2ticket_opy-w5x3d2k-OjGMlEwtTy45lB0w=Du0b/z0hSjB8im31ahGm5JAIymgon+MdXGP/2EBv7LI=\",\n",
    "  \"User-Agent\": \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36\"}\n",
    "\n",
    "data = {\n",
    "    \"token\": \"371814761\",#用户唯一的token\n",
    "    \"lang\": \"zh_CN\",\n",
    "    \"f\": \"json\",\n",
    "    \"ajax\": \"1\",\n",
    "    \"action\": \"list_ex\",\n",
    "    \"begin\": \"0\",\n",
    "    \"count\": \"5\",\n",
    "    \"query\": \"\",\n",
    "    \"fakeid\": \"MzIyODI1MzYyNA==\",#玉树芝兰的fakeid\n",
    "    \"type\": \"9\",\n",
    "}\n",
    "\n",
    "\n",
    "\n",
    "content_list=[]\n",
    "\n",
    "for i in range(5):\n",
    "    data[\"begin\"] = i*5\n",
    "    print(data[\"begin\"])\n",
    "    time.sleep(5)\n",
    "    # 使用get方法进行提交\n",
    "    content_json = requests.get(url, headers=headers, params=data).json()\n",
    "    print(content_json)\n",
    "    # 返回了一个json，里面是每一页的数据\n",
    "    for item in content_json[\"app_msg_list\"]:\n",
    "    # 提取每页文章的标题及对应的url\n",
    "        items = []\n",
    "        items.append(item[\"title\"])\n",
    "        items.append(item[\"link\"])\n",
    "        items.append(item[\"create_time\"])\n",
    "        content_list.append(items)\n",
    "\n",
    "\n",
    "name=['title','link','create_time']\n",
    "test=pd.DataFrame(columns=name,data=content_list)\n",
    "#with pd.ExcelWriter(fn[\"output\"][\"公众号_xlsx\"].format(公众号=\"玉树芝兰_requests\")) as writer:\n",
    "#    test.to_excel(writer)\n",
    "\n",
    "# test.to_csv(\"../微信公众号爬虫_zhichao/南方周末.csv\",mode='a',encoding='utf-8')\n",
    "# print(\"保存成功\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 采集公众号（selenium）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from lxml.html import fromstring\n",
    "import time\n",
    "from random import random\n",
    "\n",
    "# when selenium main_content is used\n",
    "# Parses an HTML document from a string constant.  Returns the root nood\n",
    "# root = fromstring(df.loc[1,\"html_snippets\"]) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用Selenium\n",
    "* 要更改 opts.binary_location 至自己本地的Chrome浏览器，建议portable\n",
    "* Chrome浏览器 和 chromedriver.exe要同版本号到小数后一位\n",
    "* 要确保可以 开启浏览器机器人\n",
    "* 要确保浏览器机器人 可以打开网页 driver.get(\"https://mp.weixin.qq.com\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\anaconda\\lib\\site-packages\\ipykernel_launcher.py:18: DeprecationWarning: use options instead of chrome_options\n"
     ]
    }
   ],
   "source": [
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.desired_capabilities import DesiredCapabilities\n",
    "\n",
    "#caps=dict()\n",
    "#caps[\"pageLoadStrategy\"] = \"none\"   # Do not wait for full page load\n",
    "\n",
    "opts = webdriver.ChromeOptions()\n",
    "opts.add_argument('--no-sandbox')#解决DevToolsActivePort文件不存在的报错\n",
    "opts.add_argument('window-size=1920x3000') #指定浏览器分辨率\n",
    "opts.add_argument('--disable-gpu') #谷歌文档提到需要加上一这个属性来规避bug\n",
    "opts.add_argument('--hide-scrollbars') #隐藏滚动条, 应对些特殊页面\n",
    "#opts.add_argument('blink-settings=imagesEnabled=false') #不加载图片, 提升速度\n",
    "#opts.add_argument('--headless') #浏览器不提供可视化页面. linux下如果系统不支持可视化不加这条会启动失败\n",
    "\n",
    "opts.binary_location = r\"C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe\" #\"H:\\_coding_\\Gitee\\InternetNewMedia\\CapstonePrj2016\\chromedriver.exe\"  \n",
    "\n",
    "# \"H:\\_coding_\\Gitee\\InternetNewMedia\\CapstonePrj2016\\chromedriver.exe\"  \n",
    "driver = webdriver.Chrome( chrome_options = opts) #desired_capabilities=caps,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.get(\"https://mp.weixin.qq.com\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 填表登入"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "selenium 的定位方法\n",
    "* find_element_by_id &ensp;&ensp;&ensp;  根据标签id定位\n",
    "* find_element_by_name   &ensp;&ensp;&ensp; 根据标签的name定位\n",
    "* find_element_by_xpath  &ensp;&ensp;&ensp; 根据xpath定位\n",
    "* find_element_by_link_text  &ensp;&ensp;&ensp; 通过文字链接来定位元素\n",
    "* find_element_by_partial_link_text  &ensp;&ensp;&ensp;  通过文字链接来定位元素\n",
    "* find_element_by_tag_name  &ensp;&ensp;&ensp;  根据标签的名字定位\n",
    "* find_element_by_class_name  &ensp;&ensp;&ensp; 通过class name 定位\n",
    "* find_element_by_css_selector  &ensp;&ensp;&ensp;  根据元素属性来定位"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "payload =  {\"account\": \"515695070@qq.com\", \"password\": \"********\"}\n",
    "# payload =  {\"account\": \"NFUHacks@163.com\", \"password\": \"NFU706947580\"}\n",
    "driver.find_element_by_xpath('//div[@class=\"login__type__container login__type__container__scan\"]/a').click()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "WebDriver 常用方法：\n",
    "* clear()清楚文本\n",
    "* send_keys(values)模拟按键输入\n",
    "* click()模拟点击\n",
    "* submit模拟提交"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.find_element_by_xpath('//form[@class=\"login_form\"]//input[@name=\"account\"]').clear()\n",
    "driver.find_element_by_xpath('//form[@class=\"login_form\"]//input[@name=\"account\"]').send_keys(payload['account'])\n",
    "driver.find_element_by_xpath('//form[@class=\"login_form\"]//input[@name=\"password\"]').clear()\n",
    "driver.find_element_by_xpath('//form[@class=\"login_form\"]//input[@name=\"password\"]').send_keys(payload['password'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.find_element_by_xpath('//div[@class=\"login_btn_panel\"]/a').click()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 点选单"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "其他常用方法\n",
    "* size：返回元素的尺寸\n",
    "* text：获取元素的文本\n",
    "* get_attribute：获取属性值  &ensp;&ensp;&ensp; get_attribute('innerHTML')获取元素内的全部HTML\n",
    "* is_displayed()：设置该元素用户是否可见"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'展开'"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "element = driver.find_element_by_xpath('//a[@id=\"m_open\"]')\n",
    "element.click()\n",
    "main_content = element.get_attribute('innerHTML')\n",
    "main_content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.execute_script(\"window.scrollTo(0,document.body.scrollHeight)\")#调用js execute_script方法，滚动到底部\n",
    "#https://www.sohu.com/a/337512498_400309"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://mp.weixin.qq.com/cgi-bin/appmsg?begin=0&count=10&t=media/appmsg_list&type=10&action=list&token=1226206916&lang=zh_CN'"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "element = driver.find_element_by_xpath('//li[@title[contains(.,\"素材管理\")]]/a') \n",
    "# main_content = element.get_attribute('innerHTML')\n",
    "# main_content\n",
    "url2= element.get_attribute(\"href\")\n",
    "url2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.get(url2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 新建图文消息"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "element = driver.find_element_by_xpath('//*[text()[contains(.,\"新建图文消息\")]]') \n",
    "main_content = element.get_attribute('innerHTML')\n",
    "main_content\n",
    "element.click()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['CDwindow-D784CBED556E3BB1406BA4586E2D572A', 'CDwindow-61D50EF9401D921EC812C77C29966BFB']\n"
     ]
    }
   ],
   "source": [
    "print (driver.window_handles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 新建图文消息开了另一分视窗，所以要切换 switch_to \n",
    "driver.switch_to.window(driver.window_handles[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 超链接"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                超链接              \n"
     ]
    }
   ],
   "source": [
    "element = driver.find_element_by_xpath('//*[text()[contains(.,\"超链接\")]]') \n",
    "main_content = element.get_attribute('innerHTML')\n",
    "print(main_content)\n",
    "element.click()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "选择其他公众号\n"
     ]
    }
   ],
   "source": [
    "# 点 选择其他公众号\n",
    "element = driver.find_element_by_xpath('//*[text()[contains(.,\"选择其他公众号\")]]') \n",
    "main_content = element.get_attribute('innerHTML')\n",
    "print(main_content)\n",
    "element.click()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.find_element_by_xpath('//form//div[@class=\"inner_link_account_area\"]//input[@class=\"weui-desktop-form__input\"]').clear()\n",
    "driver.find_element_by_xpath('//form//div[@class=\"inner_link_account_area\"]//input[@class=\"weui-desktop-form__input\"]').send_keys(公众号)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<div class=\"weui-desktop-icon weui-desktop-icon__inputSearch weui-desktop-icon__small\"><!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <svg width=\"16\" height=\"16\" viewBox=\"0 0 16 16\" xmlns=\"http://www.w3.org/2000/svg\"><path d=\"M11.33 10.007l4.273 4.273a.502.502 0 0 1 .005.709l-.585.584a.499.499 0 0 1-.709-.004L10.046 11.3a6.278 6.278 0 1 1 1.284-1.294zm.012-3.729a5.063 5.063 0 1 0-10.127 0 5.063 5.063 0 0 0 10.127 0z\"></path></svg> <!----> <!----> <!----> <!----></div>\n"
     ]
    }
   ],
   "source": [
    "# 点放大镜搜\n",
    "element = driver.find_element_by_xpath('//button[@class=\"weui-desktop-icon-btn weui-desktop-search__btn\"]')\n",
    "main_content = element.get_attribute('innerHTML')\n",
    "print(main_content)\n",
    "element.click()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<li class=\"inner_link_account_item\"><div class=\"weui-desktop-vm_primary\"><img src=\"http://mmbiz.qpic.cn/mmbiz_png/6fHkSrNbAAtB5gJdSibOsI9jhFcUiaPa8p43t5EIwJric3C1Hy6lBniboZpEZg0Xl2k6PsyvXZ2yQicdibQaBMYmibfAA/0?wx_fmt=png\" class=\"inner_link_account_avatar\"> <strong class=\"inner_link_account_nickname\">玉树芝兰</strong> <i class=\"inner_link_account_wechat\">微信号：nkwangshuyi</i></div> <div class=\"weui-desktop-vm_default inner_link_account_type\">订阅号</div></li><li class=\"inner_link_account_item\"><div class=\"weui-desktop-vm_primary\"><img src=\"http://mmbiz.qpic.cn/mmbiz_png/mgZvPLdW5UYYptVKktDeAfeoUAgGskZU9340DBgLDoMyvaUSjCmEibZCuYQyXlwCJkarFpKZe4ym3sLEjKJZ77A/0?wx_fmt=png\" class=\"inner_link_account_avatar\"> <strong class=\"inner_link_account_nickname\">芝兰玉树</strong> <i class=\"inner_link_account_wechat\">微信号：promisingkids</i></div> <div class=\"weui-desktop-vm_default inner_link_account_type\">服务号</div></li><li class=\"inner_link_account_item\"><div class=\"weui-desktop-vm_primary\"><img src=\"http://mmbiz.qpic.cn/mmbiz_png/5zdPAcBNnQAN0Lia7c8jyx9qYhhlmTISdWDrXzSeqtktkpCdegCnxtAGz36pj9Sc5YkRxSY54gryZxQyOCicODbQ/0?wx_fmt=png\" class=\"inner_link_account_avatar\"> <strong class=\"inner_link_account_nickname\">芝兰玉树li</strong> <i class=\"inner_link_account_wechat\">微信号：hbgsjclly</i></div> <div class=\"weui-desktop-vm_default inner_link_account_type\">订阅号</div></li><li class=\"inner_link_account_item\"><div class=\"weui-desktop-vm_primary\"><img src=\"http://mmbiz.qpic.cn/mmbiz_png/oQu4GufsxrjALoEeB5FsWOCChOOn3VPjLiaVib7GjL6RuibG5Jr7ScoaZUTuOkv1OtibSiaVI7I30rpzQphibBjdACSA/0?wx_fmt=png\" class=\"inner_link_account_avatar\"> <strong class=\"inner_link_account_nickname\">芝兰玉树2019</strong> <i class=\"inner_link_account_wechat\">微信号：zhilanyushu2019</i></div> <div class=\"weui-desktop-vm_default inner_link_account_type\">订阅号</div></li><li class=\"inner_link_account_item\"><div class=\"weui-desktop-vm_primary\"><img src=\"http://mmbiz.qpic.cn/mmbiz_png/LzWKYLw2Q0OUpaWRe8QibMMPsPicQ7lwjpkLrElXyGsZpXVfygo9nVwxxb76C3UUicrvtHKP1m2r713WficES4yNdg/0?wx_fmt=png\" class=\"inner_link_account_avatar\"> <strong class=\"inner_link_account_nickname\">芝兰玉树书画载道</strong> <i class=\"inner_link_account_wechat\">微信号：未设置</i></div> <div class=\"weui-desktop-vm_default inner_link_account_type\">订阅号</div></li>\n"
     ]
    }
   ],
   "source": [
    "element = driver.find_element_by_xpath('//ul[@class=\"inner_link_account_list\"]')\n",
    "main_content = element.get_attribute('innerHTML')\n",
    "print(main_content)\n",
    "公众号SERP = main_content\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Element div at 0x24a100de318>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 解析\n",
    "root = fromstring(公众号SERP) \n",
    "#fromstring() 可以在解抄析xml格式时，将字符串转换为Element对象，解析树的知根节点。\n",
    "root"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "主 = root.xpath('//li[@class=\"inner_link_account_item\"]')\n",
    "\n",
    "account_list = []\n",
    "for e in 主:\n",
    "    account_nickname = e.xpath('./div/strong[@class=\"inner_link_account_nickname\"]')[0].text\n",
    "    account_wechat = e.xpath('./div/i[@class=\"inner_link_account_wechat\"]')[0].text\n",
    "    account_img = e.xpath('./div/img/@src')[0]\n",
    "    account = {\"nickname\": account_nickname, \"wechat\": account_wechat, \"img\": account_img,}\n",
    "    account_list.append(account)\n",
    "\n",
    "df_account = pd.DataFrame(account_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nickname</th>\n",
       "      <th>wechat</th>\n",
       "      <th>img</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>玉树芝兰</td>\n",
       "      <td>微信号：nkwangshuyi</td>\n",
       "      <td>http://mmbiz.qpic.cn/mmbiz_png/6fHkSrNbAAtB5gJ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>芝兰玉树</td>\n",
       "      <td>微信号：promisingkids</td>\n",
       "      <td>http://mmbiz.qpic.cn/mmbiz_png/mgZvPLdW5UYYptV...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>芝兰玉树li</td>\n",
       "      <td>微信号：hbgsjclly</td>\n",
       "      <td>http://mmbiz.qpic.cn/mmbiz_png/5zdPAcBNnQAN0Li...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>芝兰玉树2019</td>\n",
       "      <td>微信号：zhilanyushu2019</td>\n",
       "      <td>http://mmbiz.qpic.cn/mmbiz_png/oQu4GufsxrjALoE...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>芝兰玉树书画载道</td>\n",
       "      <td>微信号：未设置</td>\n",
       "      <td>http://mmbiz.qpic.cn/mmbiz_png/LzWKYLw2Q0OUpaW...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   nickname               wechat  \\\n",
       "0      玉树芝兰      微信号：nkwangshuyi   \n",
       "1      芝兰玉树    微信号：promisingkids   \n",
       "2    芝兰玉树li        微信号：hbgsjclly   \n",
       "3  芝兰玉树2019  微信号：zhilanyushu2019   \n",
       "4  芝兰玉树书画载道              微信号：未设置   \n",
       "\n",
       "                                                 img  \n",
       "0  http://mmbiz.qpic.cn/mmbiz_png/6fHkSrNbAAtB5gJ...  \n",
       "1  http://mmbiz.qpic.cn/mmbiz_png/mgZvPLdW5UYYptV...  \n",
       "2  http://mmbiz.qpic.cn/mmbiz_png/5zdPAcBNnQAN0Li...  \n",
       "3  http://mmbiz.qpic.cn/mmbiz_png/oQu4GufsxrjALoE...  \n",
       "4  http://mmbiz.qpic.cn/mmbiz_png/LzWKYLw2Q0OUpaW...  "
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_account"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<div class=\"weui-desktop-vm_primary\"><img src=\"http://mmbiz.qpic.cn/mmbiz_png/6fHkSrNbAAtB5gJdSibOsI9jhFcUiaPa8p43t5EIwJric3C1Hy6lBniboZpEZg0Xl2k6PsyvXZ2yQicdibQaBMYmibfAA/0?wx_fmt=png\" class=\"inner_link_account_avatar\"> <strong class=\"inner_link_account_nickname\">玉树芝兰</strong> <i class=\"inner_link_account_wechat\">微信号：nkwangshuyi</i></div> <div class=\"weui-desktop-vm_default inner_link_account_type\">订阅号</div>\n"
     ]
    }
   ],
   "source": [
    "element = driver.find_element_by_xpath('//ul[@class=\"inner_link_account_list\"]/li')\n",
    "main_content = element.get_attribute('innerHTML')\n",
    "print(main_content)\n",
    "element.click()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\n跳转_input = driver.find_element_by_xpath(\\'//span[@class=\"weui-desktop-pagination__form\"]/input\\')\\n跳转_a = driver.find_element_by_xpath(\\'//span[@class=\"weui-desktop-pagination__form\"]/a\\')\\n跳转_input.clear()\\n跳转_input.send_keys(2)\\n跳转_a.click()\\n'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 跳转testing\n",
    "'''\n",
    "跳转_input = driver.find_element_by_xpath('//span[@class=\"weui-desktop-pagination__form\"]/input')\n",
    "跳转_a = driver.find_element_by_xpath('//span[@class=\"weui-desktop-pagination__form\"]/a')\n",
    "跳转_input.clear()\n",
    "跳转_input.send_keys(2)\n",
    "跳转_a.click()\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 41]\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "# 跳转上限\n",
    "l_e = driver.find_elements_by_xpath('//label[@class=\"weui-desktop-pagination__num\"]')\n",
    "l_e_int  = [int(x.text) for x in l_e] \n",
    "print (l_e_int)\n",
    "print (l_e_int[0]==l_e_int[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40]\n"
     ]
    }
   ],
   "source": [
    "pages = list(range(l_e_int[0],l_e_int[-1]+1 ))\n",
    "#print(pages[0:50])\n",
    "pages = list(range(41))\n",
    "print(pages)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 循环/遍历"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# global varialbes \n",
    "html_raw = dict()\n",
    "main_content =\"\"\n",
    "element = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_pages (pages):\n",
    "    for p in pages:\n",
    "        print (p,end='\\t')\n",
    "\n",
    "        跳转_input = driver.find_element_by_xpath('//span[@class=\"weui-desktop-pagination__form\"]/input')\n",
    "        跳转_a = driver.find_element_by_xpath('//span[@class=\"weui-desktop-pagination__form\"]/a')\n",
    "        跳转_input.clear()\n",
    "        跳转_input.send_keys(p)\n",
    "        跳转_a.click()\n",
    "\n",
    "        time.sleep(45+120*random())\n",
    "\n",
    "        element = driver.find_element_by_xpath('//div[@class=\"inner_link_article_list\"]')\n",
    "        main_content = element.get_attribute('innerHTML')\n",
    "        #print(main_content)\n",
    "        html_raw[p] = main_content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\t1\t2\t3\t4\t5\t6\t7\t8\t9\t10\t11\t12\t13\t14\t15\t16\t17\t18\t19\t20\t21\t22\t23\t24\t25\t26\t27\t28\t29\t30\t31\t32\t33\t34\t35\t36\t37\t38\t39\t40\t"
     ]
    }
   ],
   "source": [
    "process_pages(pages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>html_snippets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        html_snippets\n",
       "0   <div><label class=\"inner_link_article_item\"><s...\n",
       "1   <div><label class=\"inner_link_article_item\"><s...\n",
       "2   <div><label class=\"inner_link_article_item\"><s...\n",
       "3   <div><label class=\"inner_link_article_item\"><s...\n",
       "4   <div><label class=\"inner_link_article_item\"><s...\n",
       "5   <div><label class=\"inner_link_article_item\"><s...\n",
       "6   <div><label class=\"inner_link_article_item\"><s...\n",
       "7   <div><label class=\"inner_link_article_item\"><s...\n",
       "8   <div><label class=\"inner_link_article_item\"><s...\n",
       "9   <div><label class=\"inner_link_article_item\"><s...\n",
       "10  <div><label class=\"inner_link_article_item\"><s...\n",
       "11  <div><label class=\"inner_link_article_item\"><s...\n",
       "12  <div><label class=\"inner_link_article_item\"><s...\n",
       "13  <div><label class=\"inner_link_article_item\"><s...\n",
       "14  <div><label class=\"inner_link_article_item\"><s...\n",
       "15  <div><label class=\"inner_link_article_item\"><s...\n",
       "16  <div><label class=\"inner_link_article_item\"><s...\n",
       "17  <div><label class=\"inner_link_article_item\"><s...\n",
       "18  <div><label class=\"inner_link_article_item\"><s...\n",
       "19  <div><label class=\"inner_link_article_item\"><s...\n",
       "20  <div><label class=\"inner_link_article_item\"><s...\n",
       "21  <div><label class=\"inner_link_article_item\"><s...\n",
       "22  <div><label class=\"inner_link_article_item\"><s...\n",
       "23  <div><label class=\"inner_link_article_item\"><s...\n",
       "24  <div><label class=\"inner_link_article_item\"><s...\n",
       "25  <div><label class=\"inner_link_article_item\"><s...\n",
       "26  <div><label class=\"inner_link_article_item\"><s...\n",
       "27  <div><label class=\"inner_link_article_item\"><s...\n",
       "28  <div><label class=\"inner_link_article_item\"><s...\n",
       "29  <div><label class=\"inner_link_article_item\"><s...\n",
       "30  <div><label class=\"inner_link_article_item\"><s...\n",
       "31  <div><label class=\"inner_link_article_item\"><s...\n",
       "32  <div><label class=\"inner_link_article_item\"><s...\n",
       "33  <div><label class=\"inner_link_article_item\"><s...\n",
       "34  <div><label class=\"inner_link_article_item\"><s...\n",
       "35  <div><label class=\"inner_link_article_item\"><s...\n",
       "36  <div><label class=\"inner_link_article_item\"><s...\n",
       "37  <div><label class=\"inner_link_article_item\"><s...\n",
       "38  <div><label class=\"inner_link_article_item\"><s...\n",
       "39  <div><label class=\"inner_link_article_item\"><s...\n",
       "40  <div><label class=\"inner_link_article_item\"><s..."
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame([html_raw]).T\n",
    "df.columns = [\"html_snippets\"]\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stored 'html_raw' (dict)\n"
     ]
    }
   ],
   "source": [
    "%store html_raw\n",
    "import pickle #pickle模块实现了用于序列化和反序列化Python对象结构的二进制协议\n",
    "filehandler = open(\"html_raw\", 'wb') \n",
    "pickle.dump(html_raw, filehandler)#序列化对象，将对象obj保存到文件file中去"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "39\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>html_snippets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        html_snippets\n",
       "1   <div><label class=\"inner_link_article_item\"><s...\n",
       "12  <div><label class=\"inner_link_article_item\"><s..."
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_out = df[~df.duplicated()]#删除重复值\n",
    "print (len(df_out))\n",
    "df[df.duplicated()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 12]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[1, 12]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "try_again = list(df[df.duplicated()].index)\n",
    "print(try_again)\n",
    "try_again = try_again + list (set(pages).difference(set(df.index.values)))\n",
    "try_again"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 暂存档"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "#filename = fn [\"output\"] [\"公众号_htm_snippets\"] \n",
    "#df_out.to_csv(filename.format(公众号=公众号), sep=\"\\t\", encoding=\"utf8\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,"
     ]
    }
   ],
   "source": [
    "def parse_html_snippets(_snippet_):\n",
    "    root = fromstring(_snippet_) \n",
    "    title = [x.text for x in root.xpath('//div[@class=\"inner_link_article_title\"]')]\n",
    "    create_time = [x.text for x in root.xpath('//div[@class=\"inner_link_article_date\"]')]\n",
    "    link = [x for x in root.xpath('//a/@href')]\n",
    "    _df_ = pd.DataFrame({\"title\":title, \"create_time\": create_time, \"link\":link})\n",
    "    return(_df_)\n",
    "    \n",
    "l_df = []\n",
    "for p in pages:\n",
    "    _df_ = parse_html_snippets(df.loc[p,\"html_snippets\"])\n",
    "    print (len(_df_), end=\",\")\n",
    "    l_df.append(_df_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>大一就开始准备考研，合适吗？</td>\n",
       "      <td>2020-05-13</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>如何用人工智能免费帮你改英文？</td>\n",
       "      <td>2020-05-05</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>如何高效做文献回顾？（工具篇）</td>\n",
       "      <td>2020-04-28</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>卡片式笔记工具 Roam Research 介绍</td>\n",
       "      <td>2020-04-20</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>如何交互可视化你的卡片式笔记网络？</td>\n",
       "      <td>2020-04-12</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>大一就开始准备考研，合适吗？</td>\n",
       "      <td>2020-05-13</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>如何用人工智能免费帮你改英文？</td>\n",
       "      <td>2020-05-05</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>如何高效做文献回顾？（工具篇）</td>\n",
       "      <td>2020-04-28</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>卡片式笔记工具 Roam Research 介绍</td>\n",
       "      <td>2020-04-20</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>如何交互可视化你的卡片式笔记网络？</td>\n",
       "      <td>2020-04-12</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>如何用 Python 和 fast.ai 做图像深度迁移学习？</td>\n",
       "      <td>2018-10-26</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>文科生用机器学习做论文，该写些什么？</td>\n",
       "      <td>2018-10-18</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>安装 Python 软件包遇错误，怎么办？</td>\n",
       "      <td>2018-10-09</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>如何用 R 快速了解科研领域？</td>\n",
       "      <td>2018-09-25</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>《文科生数据科学上手指南》分享</td>\n",
       "      <td>2018-09-16</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>如何高效入门复杂系统仿真？</td>\n",
       "      <td>2020-02-08</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>如何用纯 Python 写 Web 应用？</td>\n",
       "      <td>2020-01-20</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>如何在朋友圈演示你的 iPhone 操作过程？</td>\n",
       "      <td>2020-01-17</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>如何不写 SQL ，探索和分析数据库？</td>\n",
       "      <td>2019-12-25</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>【福利】《巴拉巴西成功定律》抽奖赠书</td>\n",
       "      <td>2019-12-13</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>如何科学地获得成功？</td>\n",
       "      <td>2019-12-12</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              title create_time  \\\n",
       "0                    大一就开始准备考研，合适吗？  2020-05-13   \n",
       "1                   如何用人工智能免费帮你改英文？  2020-05-05   \n",
       "2                   如何高效做文献回顾？（工具篇）  2020-04-28   \n",
       "3          卡片式笔记工具 Roam Research 介绍  2020-04-20   \n",
       "4                 如何交互可视化你的卡片式笔记网络？  2020-04-12   \n",
       "5                    大一就开始准备考研，合适吗？  2020-05-13   \n",
       "6                   如何用人工智能免费帮你改英文？  2020-05-05   \n",
       "7                   如何高效做文献回顾？（工具篇）  2020-04-28   \n",
       "8          卡片式笔记工具 Roam Research 介绍  2020-04-20   \n",
       "9                 如何交互可视化你的卡片式笔记网络？  2020-04-12   \n",
       "10  如何用 Python 和 fast.ai 做图像深度迁移学习？  2018-10-26   \n",
       "11               文科生用机器学习做论文，该写些什么？  2018-10-18   \n",
       "12            安装 Python 软件包遇错误，怎么办？  2018-10-09   \n",
       "13                  如何用 R 快速了解科研领域？  2018-09-25   \n",
       "14                  《文科生数据科学上手指南》分享  2018-09-16   \n",
       "15                    如何高效入门复杂系统仿真？  2020-02-08   \n",
       "16            如何用纯 Python 写 Web 应用？  2020-01-20   \n",
       "17          如何在朋友圈演示你的 iPhone 操作过程？  2020-01-17   \n",
       "18              如何不写 SQL ，探索和分析数据库？  2019-12-25   \n",
       "19               【福利】《巴拉巴西成功定律》抽奖赠书  2019-12-13   \n",
       "20                       如何科学地获得成功？  2019-12-12   \n",
       "\n",
       "                                                 link  \n",
       "0   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "1   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "2   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "3   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "4   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "5   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "6   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "7   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "8   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "9   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "10  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "11  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "12  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "13  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "14  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "15  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "16  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "17  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "18  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "19  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "20  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  "
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_url_out = pd.concat(l_df).reset_index(drop=True)\n",
    "df_url_out.loc[0:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>200</th>\n",
       "      <td>我的MOOC之路（三）——爱恨交加机器学习</td>\n",
       "      <td>2016-07-06</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>201</th>\n",
       "      <td>Ulysses一键发送文章到Day One</td>\n",
       "      <td>2016-07-05</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>202</th>\n",
       "      <td>我的MOOC之路（二）——七周学会做搜索引擎</td>\n",
       "      <td>2016-07-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>203</th>\n",
       "      <td>微信公众号Markdown Here排版的缺失环节</td>\n",
       "      <td>2016-07-03</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>204</th>\n",
       "      <td>我的MOOC之路（一）—— 故事从这里开始</td>\n",
       "      <td>2016-07-02</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         title create_time  \\\n",
       "200      我的MOOC之路（三）——爱恨交加机器学习  2016-07-06   \n",
       "201      Ulysses一键发送文章到Day One  2016-07-05   \n",
       "202     我的MOOC之路（二）——七周学会做搜索引擎  2016-07-04   \n",
       "203  微信公众号Markdown Here排版的缺失环节  2016-07-03   \n",
       "204      我的MOOC之路（一）—— 故事从这里开始  2016-07-02   \n",
       "\n",
       "                                                  link  \n",
       "200  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "201  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "202  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "203  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "204  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  "
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_url_out.tail(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>value</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>大一就开始准备考研，合适吗？</td>\n",
       "      <td>2020-05-13</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>如何用人工智能免费帮你改英文？</td>\n",
       "      <td>2020-05-05</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>如何高效做文献回顾？（工具篇）</td>\n",
       "      <td>2020-04-28</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>大一就开始准备考研，合适吗？</td>\n",
       "      <td>2020-05-13</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>如何用人工智能免费帮你改英文？</td>\n",
       "      <td>2020-05-05</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>199</th>\n",
       "      <td>我的MOOC之路（四）——鸟群、股市与微博谣言</td>\n",
       "      <td>2016-07-11</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>201</th>\n",
       "      <td>Ulysses一键发送文章到Day One</td>\n",
       "      <td>2016-07-05</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>202</th>\n",
       "      <td>我的MOOC之路（二）——七周学会做搜索引擎</td>\n",
       "      <td>2016-07-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>203</th>\n",
       "      <td>微信公众号Markdown Here排版的缺失环节</td>\n",
       "      <td>2016-07-03</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>204</th>\n",
       "      <td>我的MOOC之路（一）—— 故事从这里开始</td>\n",
       "      <td>2016-07-02</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>125 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                           title create_time  \\\n",
       "value                                          \n",
       "0                 大一就开始准备考研，合适吗？  2020-05-13   \n",
       "1                如何用人工智能免费帮你改英文？  2020-05-05   \n",
       "2                如何高效做文献回顾？（工具篇）  2020-04-28   \n",
       "5                 大一就开始准备考研，合适吗？  2020-05-13   \n",
       "6                如何用人工智能免费帮你改英文？  2020-05-05   \n",
       "...                          ...         ...   \n",
       "199      我的MOOC之路（四）——鸟群、股市与微博谣言  2016-07-11   \n",
       "201        Ulysses一键发送文章到Day One  2016-07-05   \n",
       "202       我的MOOC之路（二）——七周学会做搜索引擎  2016-07-04   \n",
       "203    微信公众号Markdown Here排版的缺失环节  2016-07-03   \n",
       "204        我的MOOC之路（一）—— 故事从这里开始  2016-07-02   \n",
       "\n",
       "                                                    link  \n",
       "value                                                     \n",
       "0      http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "1      http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "2      http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "5      http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "6      http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "...                                                  ...  \n",
       "199    http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "201    http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "202    http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "203    http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "204    http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "\n",
       "[125 rows x 3 columns]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# tagging 标记\n",
    "tagging_list = [\"\",'数据科学','Python','数据分析','可视化','机器学习','深度学习','自然语言处理'\\\n",
    " '科研入门知识','选题','分析文献','采集数据','阅读论文','写作稿件','答辩'\\\n",
    "'效率','笔记','写作','幻灯制作','视频剪辑'\\\n",
    "'杂项','观察','思考','记录','视频', '书籍' ,'就业' '爬虫'] #overwritable\n",
    "\n",
    "v_v_list = []\n",
    "#.melt()取消将DataFrame从宽格式转换为长格式。\n",
    "for tag in tagging_list:\n",
    "    index_list = df_url_out [ df_url_out.title.str.contains(tag) ].index.tolist()\n",
    "    v_v_pairs = pd.DataFrame({tag:index_list}).melt().set_index(\"value\")\n",
    "    v_v_list.append(v_v_pairs)\n",
    "\n",
    "df_cat = v_v_list[0]\n",
    "for d in v_v_list:\n",
    "    df_cat.update(d)\n",
    "    \n",
    "# 尚未标记内容\n",
    "df_url_out.loc [ df_cat.query('variable==\"\"').index ]"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "df_url_out.loc[53].link"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>大一就开始准备考研，合适吗？</td>\n",
       "      <td>2020-05-13</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>如何用人工智能免费帮你改英文？</td>\n",
       "      <td>2020-05-05</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>如何高效做文献回顾？（工具篇）</td>\n",
       "      <td>2020-04-28</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>卡片式笔记工具 Roam Research 介绍</td>\n",
       "      <td>2020-04-20</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>如何交互可视化你的卡片式笔记网络？</td>\n",
       "      <td>2020-04-12</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>如何用 Python 和 fast.ai 做图像深度迁移学习？</td>\n",
       "      <td>2018-10-26</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>文科生用机器学习做论文，该写些什么？</td>\n",
       "      <td>2018-10-18</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>安装 Python 软件包遇错误，怎么办？</td>\n",
       "      <td>2018-10-09</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>如何用 R 快速了解科研领域？</td>\n",
       "      <td>2018-09-25</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>《文科生数据科学上手指南》分享</td>\n",
       "      <td>2018-09-16</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              title create_time  \\\n",
       "5                    大一就开始准备考研，合适吗？  2020-05-13   \n",
       "6                   如何用人工智能免费帮你改英文？  2020-05-05   \n",
       "7                   如何高效做文献回顾？（工具篇）  2020-04-28   \n",
       "8          卡片式笔记工具 Roam Research 介绍  2020-04-20   \n",
       "9                 如何交互可视化你的卡片式笔记网络？  2020-04-12   \n",
       "60  如何用 Python 和 fast.ai 做图像深度迁移学习？  2018-10-26   \n",
       "61               文科生用机器学习做论文，该写些什么？  2018-10-18   \n",
       "62            安装 Python 软件包遇错误，怎么办？  2018-10-09   \n",
       "63                  如何用 R 快速了解科研领域？  2018-09-25   \n",
       "64                  《文科生数据科学上手指南》分享  2018-09-16   \n",
       "\n",
       "                                                 link  \n",
       "5   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "6   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "7   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "8   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "9   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "60  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "61  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "62  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "63  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "64  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  "
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_url_out[df_url_out.duplicated()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>大一就开始准备考研，合适吗？</td>\n",
       "      <td>2020-05-13</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>如何用人工智能免费帮你改英文？</td>\n",
       "      <td>2020-05-05</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>如何高效做文献回顾？（工具篇）</td>\n",
       "      <td>2020-04-28</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>卡片式笔记工具 Roam Research 介绍</td>\n",
       "      <td>2020-04-20</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>如何交互可视化你的卡片式笔记网络？</td>\n",
       "      <td>2020-04-12</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200</th>\n",
       "      <td>我的MOOC之路（三）——爱恨交加机器学习</td>\n",
       "      <td>2016-07-06</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>201</th>\n",
       "      <td>Ulysses一键发送文章到Day One</td>\n",
       "      <td>2016-07-05</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>202</th>\n",
       "      <td>我的MOOC之路（二）——七周学会做搜索引擎</td>\n",
       "      <td>2016-07-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>203</th>\n",
       "      <td>微信公众号Markdown Here排版的缺失环节</td>\n",
       "      <td>2016-07-03</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>204</th>\n",
       "      <td>我的MOOC之路（一）—— 故事从这里开始</td>\n",
       "      <td>2016-07-02</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>195 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                         title create_time  \\\n",
       "0               大一就开始准备考研，合适吗？  2020-05-13   \n",
       "1              如何用人工智能免费帮你改英文？  2020-05-05   \n",
       "2              如何高效做文献回顾？（工具篇）  2020-04-28   \n",
       "3     卡片式笔记工具 Roam Research 介绍  2020-04-20   \n",
       "4            如何交互可视化你的卡片式笔记网络？  2020-04-12   \n",
       "..                         ...         ...   \n",
       "200      我的MOOC之路（三）——爱恨交加机器学习  2016-07-06   \n",
       "201      Ulysses一键发送文章到Day One  2016-07-05   \n",
       "202     我的MOOC之路（二）——七周学会做搜索引擎  2016-07-04   \n",
       "203  微信公众号Markdown Here排版的缺失环节  2016-07-03   \n",
       "204      我的MOOC之路（一）—— 故事从这里开始  2016-07-02   \n",
       "\n",
       "                                                  link  \n",
       "0    http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "1    http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "2    http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "3    http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "4    http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "..                                                 ...  \n",
       "200  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "201  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "202  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "203  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "204  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...  \n",
       "\n",
       "[195 rows x 3 columns]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_url_out[~df_url_out.duplicated()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "      <th>variable</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>大一就开始准备考研，合适吗？</td>\n",
       "      <td>2020-05-13</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>如何用人工智能免费帮你改英文？</td>\n",
       "      <td>2020-05-05</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>如何高效做文献回顾？（工具篇）</td>\n",
       "      <td>2020-04-28</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>卡片式笔记工具 Roam Research 介绍</td>\n",
       "      <td>2020-04-20</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>笔记</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>如何交互可视化你的卡片式笔记网络？</td>\n",
       "      <td>2020-04-12</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>笔记</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200</th>\n",
       "      <td>我的MOOC之路（三）——爱恨交加机器学习</td>\n",
       "      <td>2016-07-06</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>机器学习</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>201</th>\n",
       "      <td>Ulysses一键发送文章到Day One</td>\n",
       "      <td>2016-07-05</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>202</th>\n",
       "      <td>我的MOOC之路（二）——七周学会做搜索引擎</td>\n",
       "      <td>2016-07-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>203</th>\n",
       "      <td>微信公众号Markdown Here排版的缺失环节</td>\n",
       "      <td>2016-07-03</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>204</th>\n",
       "      <td>我的MOOC之路（一）—— 故事从这里开始</td>\n",
       "      <td>2016-07-02</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>205 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                         title create_time  \\\n",
       "0               大一就开始准备考研，合适吗？  2020-05-13   \n",
       "1              如何用人工智能免费帮你改英文？  2020-05-05   \n",
       "2              如何高效做文献回顾？（工具篇）  2020-04-28   \n",
       "3     卡片式笔记工具 Roam Research 介绍  2020-04-20   \n",
       "4            如何交互可视化你的卡片式笔记网络？  2020-04-12   \n",
       "..                         ...         ...   \n",
       "200      我的MOOC之路（三）——爱恨交加机器学习  2016-07-06   \n",
       "201      Ulysses一键发送文章到Day One  2016-07-05   \n",
       "202     我的MOOC之路（二）——七周学会做搜索引擎  2016-07-04   \n",
       "203  微信公众号Markdown Here排版的缺失环节  2016-07-03   \n",
       "204      我的MOOC之路（一）—— 故事从这里开始  2016-07-02   \n",
       "\n",
       "                                                  link variable  \n",
       "0    http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     无法分类  \n",
       "1    http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     无法分类  \n",
       "2    http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     无法分类  \n",
       "3    http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...       笔记  \n",
       "4    http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...       笔记  \n",
       "..                                                 ...      ...  \n",
       "200  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     机器学习  \n",
       "201  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     无法分类  \n",
       "202  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     无法分类  \n",
       "203  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     无法分类  \n",
       "204  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     无法分类  \n",
       "\n",
       "[205 rows x 4 columns]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_o = df_url_out.join(df_cat).replace(\"\", np.nan).fillna(\"无法分类\")\n",
    "df_o"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "      <th>variable</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>《文科生数据科学上手指南》分享</td>\n",
       "      <td>2018-09-16</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>数据科学</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>如何不写 SQL ，探索和分析数据库？</td>\n",
       "      <td>2019-12-25</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>如何免费获得高质量标注数据？</td>\n",
       "      <td>2019-06-12</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>如何用 Pandas 存取和交换数据？</td>\n",
       "      <td>2019-05-31</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>数据科学入门后，该做什么？</td>\n",
       "      <td>2019-05-10</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>数据科学</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>文科生 Python 与数据科学入门教材推荐</td>\n",
       "      <td>2019-04-22</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>Python</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>如何用 Python 和 Tensorflow 2.0 神经网络分类表格数据？</td>\n",
       "      <td>2019-04-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>Python</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>如何将 PDF 表格数据免费转换到 Excel ？</td>\n",
       "      <td>2019-04-01</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>如何用 Python 和 Pandas 分析犯罪记录开放数据？</td>\n",
       "      <td>2019-03-05</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>记录</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>数据科学如何助力科研？</td>\n",
       "      <td>2018-11-30</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>数据科学</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>《文科生数据科学上手指南》分享</td>\n",
       "      <td>2018-09-16</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>数据科学</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>如何用4行 R 语句，快速探索你的数据集？</td>\n",
       "      <td>2018-07-23</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75</th>\n",
       "      <td>如何在 GPU 深度学习云服务里，使用自己的数据集？</td>\n",
       "      <td>2018-07-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>深度学习</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>如何用 Python 和 API 收集与分析网络数据？</td>\n",
       "      <td>2018-05-21</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>Python</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86</th>\n",
       "      <td>如何高效入门数据科学？</td>\n",
       "      <td>2018-05-05</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>数据科学</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>如何用Python爬数据？（一）网页抓取</td>\n",
       "      <td>2018-03-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>Python</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>如何用《玉树芝兰》入门数据科学？</td>\n",
       "      <td>2018-02-22</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>数据科学</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110</th>\n",
       "      <td>如何用Python读取开放数据？</td>\n",
       "      <td>2017-12-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>Python</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>113</th>\n",
       "      <td>文科生如何高效学数据科学？</td>\n",
       "      <td>2017-11-26</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>数据科学</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>115</th>\n",
       "      <td>如何用VOSviewer分析CNKI数据？</td>\n",
       "      <td>2017-11-12</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>119</th>\n",
       "      <td>如何用R和API免费获取Web数据？</td>\n",
       "      <td>2017-10-13</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       title create_time  \\\n",
       "14                           《文科生数据科学上手指南》分享  2018-09-16   \n",
       "18                       如何不写 SQL ，探索和分析数据库？  2019-12-25   \n",
       "36                            如何免费获得高质量标注数据？  2019-06-12   \n",
       "37                       如何用 Pandas 存取和交换数据？  2019-05-31   \n",
       "40                             数据科学入门后，该做什么？  2019-05-10   \n",
       "42                    文科生 Python 与数据科学入门教材推荐  2019-04-22   \n",
       "43   如何用 Python 和 Tensorflow 2.0 神经网络分类表格数据？  2019-04-15   \n",
       "45                 如何将 PDF 表格数据免费转换到 Excel ？  2019-04-01   \n",
       "48           如何用 Python 和 Pandas 分析犯罪记录开放数据？  2019-03-05   \n",
       "56                               数据科学如何助力科研？  2018-11-30   \n",
       "64                           《文科生数据科学上手指南》分享  2018-09-16   \n",
       "73                     如何用4行 R 语句，快速探索你的数据集？  2018-07-23   \n",
       "75                如何在 GPU 深度学习云服务里，使用自己的数据集？  2018-07-04   \n",
       "83               如何用 Python 和 API 收集与分析网络数据？  2018-05-21   \n",
       "86                               如何高效入门数据科学？  2018-05-05   \n",
       "97                      如何用Python爬数据？（一）网页抓取  2018-03-04   \n",
       "98                          如何用《玉树芝兰》入门数据科学？  2018-02-22   \n",
       "110                         如何用Python读取开放数据？  2017-12-15   \n",
       "113                            文科生如何高效学数据科学？  2017-11-26   \n",
       "115                    如何用VOSviewer分析CNKI数据？  2017-11-12   \n",
       "119                       如何用R和API免费获取Web数据？  2017-10-13   \n",
       "\n",
       "                                                  link variable  \n",
       "14   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     数据科学  \n",
       "18   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     无法分类  \n",
       "36   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     无法分类  \n",
       "37   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     无法分类  \n",
       "40   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     数据科学  \n",
       "42   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...   Python  \n",
       "43   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...   Python  \n",
       "45   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     无法分类  \n",
       "48   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...       记录  \n",
       "56   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     数据科学  \n",
       "64   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     数据科学  \n",
       "73   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     无法分类  \n",
       "75   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     深度学习  \n",
       "83   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...   Python  \n",
       "86   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     数据科学  \n",
       "97   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...   Python  \n",
       "98   http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     数据科学  \n",
       "110  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...   Python  \n",
       "113  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     数据科学  \n",
       "115  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     无法分类  \n",
       "119  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     无法分类  "
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_o[df_o.title.str.contains(\"数据\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>variable</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>无法分类</th>\n",
       "      <td>125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Python</th>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>机器学习</th>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>数据科学</th>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>视频</th>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>写作</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>笔记</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>深度学习</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>可视化</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>思考</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>记录</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>选题</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          title\n",
       "variable       \n",
       "无法分类        125\n",
       "Python       38\n",
       "机器学习          9\n",
       "数据科学          7\n",
       "视频            7\n",
       "写作            5\n",
       "笔记            5\n",
       "深度学习          4\n",
       "可视化           2\n",
       "思考            1\n",
       "记录            1\n",
       "选题            1"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_stats = df_o.groupby(by=\"variable\").agg({\"title\":\"count\"}).sort_values(by=\"title\", ascending=False)\n",
    "df_stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 第二个公众号"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "公众号1 =\"Python攻城狮\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "{'app_msg_cnt': 203, 'app_msg_list': [{'aid': '2653540806_1', 'album_id': '1339195592689549313', 'appmsgid': 2653540806, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAvib7401VPHibcicChaa9Mow607g2JzoZ2aGCI9Pq3a46N6RNZnBzNa0k5tiaTj25eiapEVzKV71QjCkQQ/0?wx_fmt=jpeg', 'create_time': 1589360832, 'digest': '要让你的优秀作品在选拔中替你说话。而好的作品，需要时光的雕刻。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540806&idx=1&sn=2e40ec2d254c63d1c95856208814adfd&chksm=f389bb11c4fe3207d9269f229588ff7d592f5268616e9f91e4506c72655af8d0488738f78f1e#rd', 'tagid': [], 'title': '大一就开始准备考研，合适吗？', 'update_time': 1589360832}, {'aid': '2653540789_1', 'album_id': '1339188083526434816', 'appmsgid': 2653540789, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAtfds6VFs8mM2EI4tv51z43ySzmzOveFnwq9Pia3G1UOIsm1Zht1kZUCtzG9p9AicMWnAsia9qGFstmw/0?wx_fmt=jpeg', 'create_time': 1588674888, 'digest': '推荐给你一款最新的免费工具，解决你英文改写润色的后顾之忧。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540789&idx=1&sn=d2562da2dc5e86a69ae16a028c02915b&chksm=f389bb62c4fe32749c23fd6d70c8a4fcbd10e3969c5e6eb2b5f6e86bf8bc5d19ac7e374981e3#rd', 'tagid': [], 'title': '如何用人工智能免费帮你改英文？', 'update_time': 1588674888}, {'aid': '2653540778_1', 'album_id': '1339183650063220738', 'appmsgid': 2653540778, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAthWnZzHYEHEYo12EcrOmBHr3LFpsia0ybuVJXLsowiaYOWaJZ3zc8BeOQeAttZ31TSicaBlTiaNdABtQ/0?wx_fmt=jpeg', 'create_time': 1588072166, 'digest': '怎样把琐碎的「文献采集」、「标注笔记」、「整合引用」，和「样式输出」一气呵成地自动化搞定？', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 1, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540778&idx=1&sn=b614b519ac2e30ece6a020ee9a2cf716&chksm=f389bb7dc4fe326b179498bee21774e7f0b10b3c761e4661944c530fcfc2f7f378d2f947340d#rd', 'tagid': [], 'title': '如何高效做文献回顾？（工具篇）', 'update_time': 1588072166}, {'aid': '2653540768_1', 'album_id': '0', 'appmsgid': 2653540768, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAstPZ1OqLj8kx9CSiafv9GVNuD4yjK93JichSBIUcECBGMaJRiag3UrrXSZ12d3dHQ9F2yaNcy4m768w/0?wx_fmt=jpeg', 'create_time': 1587391737, 'digest': '「工欲善其事，必先利其器。」想要用好卡片式笔记进行学习与创作，没有一款趁手的工具怎么行？这一讲，我为你介绍卡片式笔记工具 Roam Research 的一些基本功能。希望你能感受到我第一次看到它时的惊艳。', 'has_red_packet_cover': 0, 'is_pay_subscribe': 0, 'item_show_type': 5, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540768&idx=1&sn=f4cdff85383be13c41662f0ab4f522b6&chksm=f389bb77c4fe32611d4666dbb30985bd01accdf354319721cbddc0ab50a46f72052f5c65ff5e#rd', 'tagid': [], 'title': '卡片式笔记工具 Roam Research 介绍', 'update_time': 1587391737}, {'aid': '2653540764_1', 'album_id': '1339188083526434816', 'appmsgid': 2653540764, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAvYf0RttvmH3uNia5Zbr3iahr4CycibA9ecma8vLvS6K9Ks5klB8zquMuveiaZfF5BT7qr0OxsjhKB4Lw/0?wx_fmt=jpeg', 'create_time': 1586647921, 'digest': '会当凌绝顶，一览众山小。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540764&idx=1&sn=b192edb5b54b2c04f814fa1b4e94bf02&chksm=f389bb4bc4fe325dbd969500dce4f9efff1217de584340a4a2a5e974078b08d377533b1f7aa4#rd', 'tagid': [], 'title': '如何交互可视化你的卡片式笔记网络？', 'update_time': 1586647921}], 'base_resp': {'err_msg': 'ok', 'ret': 0}}\n",
      "5\n",
      "{'app_msg_cnt': 203, 'app_msg_list': [{'aid': '2653540753_1', 'album_id': '0', 'appmsgid': 2653540753, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAsmNBmKdWVaZiaG5QVru1MY3b0C9XjPMvHsPy7GVVDohXCrSFYhqUDItiaDCSXgaHVgzvfVNUp8FDcw/0?wx_fmt=jpeg', 'create_time': 1585188990, 'digest': '你好！我是王树义老师。最近流行线上教学。有位朋友准备尝试把线下培训转移到线上，希望我能给提出一些建议。我于是做了这个分享。如果你正准备开始线上教学，希望这个视频能帮你少踩一些坑。如果你已经有了一段时间的教学经验，我也希望和你切磋交流，一起提升咱们的线上教学质量。如果你对本视频讲述的内容有疑问，或者要提出批评建议，欢迎留言。', 'has_red_packet_cover': 0, 'is_pay_subscribe': 0, 'item_show_type': 5, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540753&idx=1&sn=a0cdbe7c918e83e34b18881ebe50f272&chksm=f389bb46c4fe32508a6981d4fe0502905bd9a9f8900501bf5f9d454345f7ebcc7228e1a5b33f#rd', 'tagid': [], 'title': '线上教学心得', 'update_time': 1585188990}, {'aid': '2653540745_1', 'album_id': '1339195592689549313', 'appmsgid': 2653540745, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAvEcelEQLnnIuJCbeIpLpjO67Mf5yKeLFgt4bfHwwibMjOJFDtd64AmhMeg6H6z51WLBXB6tYJoia6g/0?wx_fmt=jpeg', 'create_time': 1584072711, 'digest': '我有一个坏消息，和一个好消息，你想先听哪个？', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540745&idx=1&sn=e0f4f6224834dce897bfb9a441334aef&chksm=f389bb5ec4fe32487faa3b35ac2bbe55b72327ac3a46c7dc41b65e97fc330f1d23d4e21d9ef8#rd', 'tagid': [], 'title': '扩招了，你的研究生学历会不会贬值？', 'update_time': 1584072711}, {'aid': '2653540740_1', 'album_id': '1339183650063220738', 'appmsgid': 2653540740, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAuUPZbgcjCYwib0v1I3BZKOXnLUiaPAAKBlAYXYBiaibzfMtXBKZxTSGI71ricRs0FpH1c26IUX4ibChErA/0?wx_fmt=jpeg', 'create_time': 1583760290, 'digest': '横看成岭侧成峰，远近高低各不同。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540740&idx=1&sn=67869ae1f36c45f06c637e07fe775a2f&chksm=f389bb53c4fe32451762eafdd83c7699245dc73d60f95080f3fe0d6220f511e9ad1ee0fe5bf6#rd', 'tagid': [], 'title': '如何用卡片法写论文？', 'update_time': 1583760290}, {'aid': '2653540733_1', 'album_id': '1339188083526434816', 'appmsgid': 2653540733, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAuiaE6hDlry6ic21BvLaxgbF131bzPyUEs0fSNYsdqDl7TO279qhJVDKFMoVG0ov3vN8BqicaW9qdl7w/0?wx_fmt=jpeg', 'create_time': 1582636402, 'digest': '这种诸多写作高手推荐的方法，实践起来为何这么困难？咱们来聊聊。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540733&idx=1&sn=2d3ccf04a0086c6137b06c731e0805c4&chksm=f389bbaac4fe32bc2c7e565dbfe954474dc2f16d884b0f6743f2adfe3447b0109fd74901b941#rd', 'tagid': [], 'title': '如何高效实践卡片式写作？', 'update_time': 1582636402}, {'aid': '2653540728_1', 'album_id': '0', 'appmsgid': 2653540728, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAuY2VrwcyA28icUeGicYLMkicbjhUI4VdcloicMTNBDHOB5nqeyp5dGrNibjt996qagib5GjoJ5nXuE8UcQ/0?wx_fmt=jpeg', 'create_time': 1581322334, 'digest': '希望这些推荐，可以帮助你更加安心、健康地宅在家里。为抗击疫情、阻断病毒、安定社会，做出自己的一份贡献。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540728&idx=1&sn=339f2e2406426c5e79ad855a6551ad26&chksm=f389bbafc4fe32b9f5982782aec37ed36757f8d3bca9c773576f6ae29cf225c6a77fbcff63cd#rd', 'tagid': [], 'title': '抗疫宅家，给你推荐几个隔离工具', 'update_time': 1581322334}], 'base_resp': {'err_msg': 'ok', 'ret': 0}}\n",
      "10\n",
      "{'app_msg_cnt': 203, 'app_msg_list': [{'aid': '2653540724_1', 'album_id': '0', 'appmsgid': 2653540724, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAtOdOLWAm4zzNeD9AmV9Lche5Tw5AlTQDAPotejd1o0iaJuAvyVnsNKWNRb1yzC6LlmaDaAL4yMDxQ/0?wx_fmt=jpeg', 'create_time': 1581150496, 'digest': '推荐给你一门好课。帮你在研究方法武器库中，添上复杂系统仿真这一项。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540724&idx=1&sn=713050c012d7724bf8a3240991b05859&chksm=f389bba3c4fe32b54f27599185bfd09d31025a86f78f585c10c95a6ada96d663a8a44e58b3ed#rd', 'tagid': [], 'title': '如何高效入门复杂系统仿真？', 'update_time': 1581150496}, {'aid': '2653540719_1', 'album_id': '1339178811027587075', 'appmsgid': 2653540719, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAtjvnxO9nPT8iaAeibiaG6B4iaWgfoNdT2ZDoSdrkzqibDrxGActYVlG11LbLpiaLdpm13XhsmjbXSqXD7Q/0?wx_fmt=jpeg', 'create_time': 1579502752, 'digest': '不用学前端编程，你就能用 Python 简单高效写出漂亮的交互式 Web 应用，将你的数据分析成果立即展示给团队和客户。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540719&idx=1&sn=226b8ffaa92d7b21a81d3d3bda28ba6b&chksm=f389bbb8c4fe32ae1c192fa5c2888b54057b6ecb02bf63c20ea5930fa204c792bf33b24f0fbd#rd', 'tagid': [], 'title': '如何用纯 Python 写 Web 应用？', 'update_time': 1579502752}, {'aid': '2653540714_1', 'album_id': '1339188083526434816', 'appmsgid': 2653540714, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAtdgrJqYOuIMA2ogE2OUrhYrKibN7tBFtzGRM0vCDHHEFe9thLHkiaQ4beRxHtNwF8ZmicKPmptxWBMA/0?wx_fmt=jpeg', 'create_time': 1579265586, 'digest': '想把 iPhone 上的操作流程录制下来，通过朋友圈分享给小伙伴？这几个小技巧可能对你有用。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540714&idx=1&sn=bfa3615317cfaf371386a52312d4f059&chksm=f389bbbdc4fe32ab00d97bd1b34bfad67115afa995d76797818a359bfa4c2aaa193312c2fd5f#rd', 'tagid': [], 'title': '如何在朋友圈演示你的 iPhone 操作过程？', 'update_time': 1579265586}, {'aid': '2653540707_1', 'album_id': '1339188083526434816', 'appmsgid': 2653540707, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAsDgZVNXkeEcIXHCDqR8MUSqCj5UnTvVWuicHE18NyLovYJia5KowoR0hqL7O4QJYCcHuBfyZM19jlg/0?wx_fmt=jpeg', 'create_time': 1577197630, 'digest': '数据分析的门槛在降低，希望你能及时意识到。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540707&idx=1&sn=5253093da915156d28c9f5bb91d957e1&chksm=f389bbb4c4fe32a25b71b802613b972ae9514ac935092630820fc985f56e782074331833f9dc#rd', 'tagid': [], 'title': '如何不写 SQL ，探索和分析数据库？', 'update_time': 1577226000}, {'aid': '2653540699_1', 'album_id': '0', 'appmsgid': 2653540699, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAvYGfv1DQGBO3JIeLWGNj908uJQHwUqmZw1GbuicsmoyhHkgkGOs5ySoptjwP9x1SI324W79APia6Ig/0?wx_fmt=jpeg', 'create_time': 1576208645, 'digest': '祝好运气！', 'has_red_packet_cover': 0, 'is_original': 0, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540699&idx=1&sn=b4cc0ea725350b93d14ee3eb0b5d9180&chksm=f389bb8cc4fe329ac6289b9fa74827278d0f84fbc81e67e655661e8762cdc515af23eb0b39d9#rd', 'tagid': [], 'title': '【福利】《巴拉巴西成功定律》抽奖赠书', 'update_time': 1576208645}], 'base_resp': {'err_msg': 'ok', 'ret': 0}}\n",
      "15\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'app_msg_cnt': 203, 'app_msg_list': [{'aid': '2653540692_1', 'album_id': '0', 'appmsgid': 2653540692, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAvrnl9s5GfM5HsDXFibP1KmjcjEic8HFPnxlca3gAawQCTs7VBZwTOicNzw5ic2Dt14TESMfibB8ickXusA/0?wx_fmt=jpeg', 'create_time': 1576159121, 'digest': '给你推荐一本“武器级”图书。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540692&idx=1&sn=21c296472df734f2d82f30b9e9af04d8&chksm=f389bb83c4fe3295bfed33191fe4f48e673034a2dc86b9b2d4082eebc24a69558f691cc678ef#rd', 'tagid': [], 'title': '如何科学地获得成功？', 'update_time': 1576159121}, {'aid': '2653540688_1', 'album_id': '0', 'appmsgid': 2653540688, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAtyI6hsg2YdqaAsZAZ7dEX846jf0a9r4yoljLh21L88Mia81E04zXcfbjz7NkQ6nDKPO1viah8X571Q/0?wx_fmt=jpeg', 'create_time': 1575895803, 'digest': 'PyTorch 入门，坑着实不少。咱们来谈谈，如何选个合适的教程，避开它们。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540688&idx=1&sn=668e9a979b79dc589c6e5b74aff92bcd&chksm=f389bb87c4fe3291d4ff1f1e34178362e91273e2ce3849da546ae6743039877acef6a8da50dc#rd', 'tagid': [], 'title': '如何高效入门 PyTorch ？', 'update_time': 1575895803}, {'aid': '2653540684_1', 'album_id': '1339188083526434816', 'appmsgid': 2653540684, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAs95Eib8kupxoa1Q3vnvWuCsV45tBDeVbTnibugUPaockJZgLgyhGpiaQAA1QboSkaHVbbIswmCWes1g/0?wx_fmt=jpeg', 'create_time': 1574072422, 'digest': '毕业论文字数要求太多，写起来很苦恼？把我新发现的写作利器分享给你。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540684&idx=1&sn=36a64c276264680fa54f060970118f94&chksm=f389bb9bc4fe328d4bc245718e086a40b9ba27ace93e967bc9de281d06802264f88f8a8b60dd#rd', 'tagid': [], 'title': '如何高效写长文？', 'update_time': 1574072422}, {'aid': '2653540680_1', 'album_id': '1339183650063220738', 'appmsgid': 2653540680, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAsN7r4UzicwNR7Np7P7UD8iamghsDBvQlkTDpdl0eTOSbYqYVlBhbibBBMSmwA7AU77Bhtv2Jraq21mg/0?wx_fmt=jpeg', 'create_time': 1572915307, 'digest': '悄悄告诉你几个窍门。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540680&idx=1&sn=6dfb8f84507c5c974f4a3eb8300ba22c&chksm=f389bb9fc4fe328931fe1193d4819d2cda36e9af82f0aac39a9913b8705ec7df439c944fadd3#rd', 'tagid': [], 'title': '论文读不懂怎么办？', 'update_time': 1572915307}, {'aid': '2653540676_1', 'album_id': '1339183650063220738', 'appmsgid': 2653540676, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAscVvTI2lJdDAgzWtaBmgeBYHqd7NoJctbqMcwtC5WDNE07gibMpsmH4r8kHktymEXCFQmgt6WkL6w/0?wx_fmt=jpeg', 'create_time': 1571644554, 'digest': '用产品经理的视角，找寻论文专业读者的知识需求。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540676&idx=1&sn=7ff14b24e2a1987141385b2dd4e6a738&chksm=f389bb93c4fe32859369249715b6104144bea7a364122cc58a0daad9504c79d34e74c993ddf8#rd', 'tagid': [], 'title': '如何与导师有效沟通你的论文选题？', 'update_time': 1571644554}], 'base_resp': {'err_msg': 'ok', 'ret': 0}}\n",
      "20\n",
      "{'app_msg_cnt': 203, 'app_msg_list': [{'aid': '2653540672_1', 'album_id': '1339188083526434816', 'appmsgid': 2653540672, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAtD2h49twibLXhsYVDZPLDCicAqn9ib9LpG0bBzUzY29VuYlVJ6TT1Z220A6muiay31rmE61K3iaiakRQqA/0?wx_fmt=jpeg', 'create_time': 1570284102, 'digest': '工欲善其事，必先利其器。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540672&idx=1&sn=5be1b09e9ea2e07d1fa9eec72d83e806&chksm=f389bb97c4fe3281c2d3826ab197dc5780bfac5651815e30b67efdd9aa3417fde71c05868b10#rd', 'tagid': [], 'title': '如何高效写作公众号文章？', 'update_time': 1570284101}, {'aid': '2653540664_1', 'album_id': '1339178811027587075', 'appmsgid': 2653540664, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAsN0LR2icSqBE9DykHibtCCiaRbwXTMXGd8d4cjmcNa4B8TOnPqM5XDpqVH9URdObxIrD5Wpqut9azCw/0?wx_fmt=jpeg', 'create_time': 1569423942, 'digest': '简单、直接、高效地从文本中提炼关键内容。让你体验正则表达式的威力。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540664&idx=1&sn=fbd0088427d8133b80964e010df2494d&chksm=f389bbefc4fe32f9a2b1255779e67cbfb49259a6d43e1261d94abcc0ae4ea8adef87719038cd#rd', 'tagid': [], 'title': '如何用 Python 和正则表达式抽取文本结构化信息？', 'update_time': 1569450600}, {'aid': '2653540660_1', 'album_id': '1339183650063220738', 'appmsgid': 2653540660, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAvQZ7b6ejaQbvcluGp4oib1MHejsELabnqXs90DMgnC1loWt2bT0KDZsqFg6vItJFzptKoxYDgPhicg/0?wx_fmt=jpeg', 'create_time': 1568599343, 'digest': '四大心法，助你写出更好的“第一封邮件”。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540660&idx=1&sn=fef006a155a52385c124394b5187d48d&chksm=f389bbe3c4fe32f5c91ab7c4fcbe52a5a02a6cf4b17130626f9163253302de16ee453c08be57#rd', 'tagid': [], 'title': '给导师的套磁信该怎么写？', 'update_time': 1568599343}, {'aid': '2653540656_1', 'album_id': '0', 'appmsgid': 2653540656, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAt3jsNzUCrtKsF92icYqOOQkvVYePBhKftLceNjNXFpqVpjibXmnwH9zPkPXDqahHBnfrQEZc9j42IQ/0?wx_fmt=jpeg', 'create_time': 1567952134, 'digest': '解决眼高手低，促进触类旁通。', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540656&idx=1&sn=336e3c9284b15cf24c38d063468ed79e&chksm=f389bbe7c4fe32f1df04ae208ab4350d95503db8deeff3e8c44b3be164af53823f47067e2d6c#rd', 'tagid': [], 'title': '“没有灵魂”的AI作诗，有啥用？', 'update_time': 1567952133}, {'aid': '2653540651_1', 'album_id': '0', 'appmsgid': 2653540651, 'checking': 0, 'cover': 'https://mmbiz.qlogo.cn/mmbiz_jpg/6fHkSrNbAAtGjgSicRCRsty9hpY7eQhgGTJMibOcK6d4GkjRUOw26sQhZ7Emzic0rvA75wN2lC8wga9zAl7HmYrAw/0?wx_fmt=jpeg', 'create_time': 1566999097, 'digest': '听好了，你只有两个选项……', 'has_red_packet_cover': 0, 'is_original': 1, 'is_pay_subscribe': 0, 'item_show_type': 0, 'itemidx': 1, 'link': 'http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA==&mid=2653540651&idx=1&sn=5f729b88363d17ca199e10906756331a&chksm=f389bbfcc4fe32eac8cbb14e7b4df586914cc823fa10a160f4c19edc0c8ad450cf3af5be738c#rd', 'tagid': [], 'title': '美国人喜欢工会吗？', 'update_time': 1566999097}], 'base_resp': {'err_msg': 'ok', 'ret': 0}}\n"
     ]
    }
   ],
   "source": [
    "\n",
    "data = {\n",
    "    \"token\": \"371814761\",#用户唯一的token\n",
    "    \"lang\": \"zh_CN\",\n",
    "    \"f\": \"json\",\n",
    "    \"ajax\": \"1\",\n",
    "    \"action\": \"list_ex\",\n",
    "    \"begin\": \"0\",\n",
    "    \"count\": \"5\",\n",
    "    \"query\": \"\",\n",
    "    \"fakeid\": \"MzIyODI1MzYyNA==\",#玉树芝兰的fakeid\n",
    "    \"type\": \"9\",\n",
    "}\n",
    "\n",
    "\n",
    "content_list=[]\n",
    "\n",
    "for i in range(5):\n",
    "    data[\"begin\"] = i*5\n",
    "    print(data[\"begin\"])\n",
    "    time.sleep(5)\n",
    "    # 使用get方法进行提交\n",
    "    content_json = requests.get(url, headers=headers, params=data).json()\n",
    "    print(content_json)\n",
    "    # 返回了一个json，里面是每一页的数据\n",
    "    for item in content_json[\"app_msg_list\"]:\n",
    "    # 提取每页文章的标题及对应的url\n",
    "        items = []\n",
    "        items.append(item[\"title\"])\n",
    "        items.append(item[\"link\"])\n",
    "        items.append(item[\"create_time\"])\n",
    "        content_list.append(items)\n",
    "\n",
    "\n",
    "name=['title','link','create_time']\n",
    "test1=pd.DataFrame(columns=name,data=content_list)\n",
    "with pd.ExcelWriter(fn[\"output\"][\"公众号_xlsx\"].format(公众号=\"Python攻城狮_requests\")) as writer:\n",
    "    test1.to_excel(writer)   \n",
    "#writer = pd.ExcelWriter(r'E:\\2019-2020\\web数据挖掘\\wk9\\data_sets\\公众号_url_玉树芝兰.xlsx')\n",
    "#test1.to_excel(writer, sheet_name=\"Python攻城狮\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "选择其他公众号\n"
     ]
    }
   ],
   "source": [
    "# 点 选择其他公众号\n",
    "element = driver.find_element_by_xpath('//*[text()[contains(.,\"选择其他公众号\")]]') \n",
    "main_content = element.get_attribute('innerHTML')\n",
    "print(main_content)\n",
    "element.click()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.find_element_by_xpath('//form//div[@class=\"inner_link_account_area\"]//input[@class=\"weui-desktop-form__input\"]').clear()\n",
    "driver.find_element_by_xpath('//form//div[@class=\"inner_link_account_area\"]//input[@class=\"weui-desktop-form__input\"]').send_keys(公众号1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<div class=\"weui-desktop-icon weui-desktop-icon__inputSearch weui-desktop-icon__small\"><!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <svg width=\"16\" height=\"16\" viewBox=\"0 0 16 16\" xmlns=\"http://www.w3.org/2000/svg\"><path d=\"M11.33 10.007l4.273 4.273a.502.502 0 0 1 .005.709l-.585.584a.499.499 0 0 1-.709-.004L10.046 11.3a6.278 6.278 0 1 1 1.284-1.294zm.012-3.729a5.063 5.063 0 1 0-10.127 0 5.063 5.063 0 0 0 10.127 0z\"></path></svg> <!----> <!----> <!----> <!----></div>\n"
     ]
    }
   ],
   "source": [
    "# 点放大镜搜\n",
    "element = driver.find_element_by_xpath('//button[@class=\"weui-desktop-icon-btn weui-desktop-search__btn\"]')\n",
    "main_content = element.get_attribute('innerHTML')\n",
    "print(main_content)\n",
    "element.click()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Element div at 0x24a11429638>"
      ]
     },
     "execution_count": 174,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 解析\n",
    "root1= fromstring(公众号SERP1) \n",
    "#fromstring() 可以在解抄析xml格式时，将字符串转换为Element对象，解析树的知根节点。\n",
    "root1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<li class=\"inner_link_account_item\"><div class=\"weui-desktop-vm_primary\"><img src=\"http://mmbiz.qpic.cn/mmbiz_png/aSJyGw23SZCcbRDgW1cQFRJ3te1PPzcF9fTuSWC1kTzicvZwCSGjqPdN3VkicxzCYR6g2cBJlAYiaEibOSlDBjvxWQ/0?wx_fmt=png\" class=\"inner_link_account_avatar\"> <strong class=\"inner_link_account_nickname\">Python攻城狮</strong> <i class=\"inner_link_account_wechat\">微信号：未设置</i></div> <div class=\"weui-desktop-vm_default inner_link_account_type\">订阅号</div></li><li class=\"inner_link_account_item\"><div class=\"weui-desktop-vm_primary\"><img src=\"http://mmbiz.qpic.cn/mmbiz_png/8vaxOIpAdmuCxJUn8Ezr432caqmpBWQDWcxtMj5iaHoSFLyQqTKxBIDE0Wib5edhSW4DsYM9T7QW7qWo7ia5aBibuw/0?wx_fmt=png\" class=\"inner_link_account_avatar\"> <strong class=\"inner_link_account_nickname\">python攻城狮基地</strong> <i class=\"inner_link_account_wechat\">微信号：未设置</i></div> <div class=\"weui-desktop-vm_default inner_link_account_type\">订阅号</div></li>\n"
     ]
    }
   ],
   "source": [
    "element = driver.find_element_by_xpath('//ul[@class=\"inner_link_account_list\"]')\n",
    "main_content1 = element.get_attribute('innerHTML')\n",
    "print(main_content1)\n",
    "公众号SERP1 = main_content1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Element li at 0x24a11429778>, <Element li at 0x24a114297c8>]"
      ]
     },
     "execution_count": 176,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "主1 = root1.xpath('//li[@class=\"inner_link_account_item\"]')\n",
    "主1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nickname</th>\n",
       "      <th>wechat</th>\n",
       "      <th>img</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>芝兰玉树书画载道</td>\n",
       "      <td>微信号：未设置</td>\n",
       "      <td>http://mmbiz.qpic.cn/mmbiz_png/LzWKYLw2Q0OUpaW...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>芝兰玉树书画载道</td>\n",
       "      <td>微信号：未设置</td>\n",
       "      <td>http://mmbiz.qpic.cn/mmbiz_png/LzWKYLw2Q0OUpaW...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   nickname   wechat                                                img\n",
       "0  芝兰玉树书画载道  微信号：未设置  http://mmbiz.qpic.cn/mmbiz_png/LzWKYLw2Q0OUpaW...\n",
       "1  芝兰玉树书画载道  微信号：未设置  http://mmbiz.qpic.cn/mmbiz_png/LzWKYLw2Q0OUpaW..."
      ]
     },
     "execution_count": 189,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#主 = root.xpath('//li[@class=\"inner_link_account_item\"]')\n",
    "\n",
    "account_list1 = []\n",
    "for i in 主1:\n",
    "    account_nickname1 = e.xpath('./div/strong[@class=\"inner_link_account_nickname\"]')[0].text\n",
    "    account_wechat1 = e.xpath('./div/i[@class=\"inner_link_account_wechat\"]')[0].text\n",
    "    account_img1 = e.xpath('./div/img/@src')[0]\n",
    "    account1 = {\"nickname\": account_nickname1, \"wechat\": account_wechat1, \"img\": account_img1,}\n",
    "    account_list1.append(account1)\n",
    "\n",
    "df_account1= pd.DataFrame(account_list1)\n",
    "df_account1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<div class=\"weui-desktop-vm_primary\"><img src=\"http://mmbiz.qpic.cn/mmbiz_png/aSJyGw23SZCcbRDgW1cQFRJ3te1PPzcF9fTuSWC1kTzicvZwCSGjqPdN3VkicxzCYR6g2cBJlAYiaEibOSlDBjvxWQ/0?wx_fmt=png\" class=\"inner_link_account_avatar\"> <strong class=\"inner_link_account_nickname\">Python攻城狮</strong> <i class=\"inner_link_account_wechat\">微信号：未设置</i></div> <div class=\"weui-desktop-vm_default inner_link_account_type\">订阅号</div>\n"
     ]
    }
   ],
   "source": [
    "element= driver.find_element_by_xpath('//ul[@class=\"inner_link_account_list\"]/li')\n",
    "main_content1= element.get_attribute('innerHTML')\n",
    "print(main_content1)\n",
    "element.click()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[28, 29]\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "# 跳转上限\n",
    "l_e = driver.find_elements_by_xpath('//label[@class=\"weui-desktop-pagination__num\"]')\n",
    "l_e_int  = [int(x.text) for x in l_e] \n",
    "print (l_e_int)\n",
    "print (l_e_int[0]==l_e_int[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28]\n"
     ]
    }
   ],
   "source": [
    "pages = list(range(l_e_int[0],l_e_int[-1]+1 ))\n",
    "#print(pages[0:50])\n",
    "pages = list(range(29))\n",
    "print(pages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [],
   "source": [
    "# global varialbes \n",
    "html_raw = dict()\n",
    "main_content =\"\"\n",
    "element = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_pages1 (pages):\n",
    "    time.sleep(0.5)\n",
    "    for p in pages:\n",
    "        print (p,end='\\t')\n",
    "\n",
    "        跳转_input = driver.find_element_by_xpath('//span[@class=\"weui-desktop-pagination__form\"]/input')\n",
    "        跳转_a = driver.find_element_by_xpath('//span[@class=\"weui-desktop-pagination__form\"]/a')\n",
    "        跳转_input.clear()\n",
    "        跳转_input.send_keys(p)\n",
    "        跳转_a.click()\n",
    "\n",
    "        time.sleep(45+120*random())\n",
    "\n",
    "        element = driver.find_element_by_xpath('//div[@class=\"inner_link_article_list\"]')\n",
    "        main_content = element.get_attribute('innerHTML')\n",
    "        #print(main_content)\n",
    "        html_raw[p] = main_content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\t1\t2\t3\t4\t5\t6\t7\t8\t9\t10\t11\t12\t13\t14\t15\t16\t17\t18\t19\t20\t21\t22\t23\t24\t25\t26\t27\t28\t"
     ]
    }
   ],
   "source": [
    "process_pages1(pages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>html_snippets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        html_snippets\n",
       "0   <div><label class=\"inner_link_article_item\"><s...\n",
       "1   <div><label class=\"inner_link_article_item\"><s...\n",
       "2   <div><label class=\"inner_link_article_item\"><s...\n",
       "3   <div><label class=\"inner_link_article_item\"><s...\n",
       "4   <div><label class=\"inner_link_article_item\"><s...\n",
       "5   <div><label class=\"inner_link_article_item\"><s...\n",
       "6   <div><label class=\"inner_link_article_item\"><s...\n",
       "7   <div><label class=\"inner_link_article_item\"><s...\n",
       "8   <div><label class=\"inner_link_article_item\"><s...\n",
       "9   <div><label class=\"inner_link_article_item\"><s...\n",
       "10  <div><label class=\"inner_link_article_item\"><s...\n",
       "11  <div><label class=\"inner_link_article_item\"><s...\n",
       "12  <div><label class=\"inner_link_article_item\"><s...\n",
       "13  <div><label class=\"inner_link_article_item\"><s...\n",
       "14  <div><label class=\"inner_link_article_item\"><s...\n",
       "15  <div><label class=\"inner_link_article_item\"><s...\n",
       "16  <div><label class=\"inner_link_article_item\"><s...\n",
       "17  <div><label class=\"inner_link_article_item\"><s...\n",
       "18  <div><label class=\"inner_link_article_item\"><s...\n",
       "19  <div><label class=\"inner_link_article_item\"><s...\n",
       "20  <div><label class=\"inner_link_article_item\"><s...\n",
       "21  <div><label class=\"inner_link_article_item\"><s...\n",
       "22  <div><label class=\"inner_link_article_item\"><s...\n",
       "23  <div><label class=\"inner_link_article_item\"><s...\n",
       "24  <div><label class=\"inner_link_article_item\"><s...\n",
       "25  <div><label class=\"inner_link_article_item\"><s...\n",
       "26  <div><label class=\"inner_link_article_item\"><s...\n",
       "27  <div><label class=\"inner_link_article_item\"><s...\n",
       "28  <div><label class=\"inner_link_article_item\"><s..."
      ]
     },
     "execution_count": 147,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1 = pd.DataFrame([html_raw]).T\n",
    "df1.columns = [\"html_snippets\"]\n",
    "df1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stored 'html_raw' (dict)\n"
     ]
    }
   ],
   "source": [
    "%store html_raw\n",
    "import pickle #pickle模块实现了用于序列化和反序列化Python对象结构的二进制协议\n",
    "filehandler = open(\"html_raw1\", 'wb') \n",
    "pickle.dump(html_raw, filehandler)#序列化对象，将对象obj保存到文件file中去"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "27\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>html_snippets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        html_snippets\n",
       "1   <div><label class=\"inner_link_article_item\"><s...\n",
       "12  <div><label class=\"inner_link_article_item\"><s..."
      ]
     },
     "execution_count": 149,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1_out = df1[~df1.duplicated()]#删除重复值\n",
    "print (len(df1_out))\n",
    "df1[df1.duplicated()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [],
   "source": [
    "#filename = fn [\"output\"] [\"公众号_htm_snippets\"] \n",
    "#df1_out.to_csv(filename.format(公众号=公众号), sep=\"\\t\", encoding=\"utf8\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6,6,6,5,7,5,5,5,5,5,5,5,6,5,5,5,5,5,5,5,6,5,9,7,6,11,5,15,6,"
     ]
    }
   ],
   "source": [
    "def parse_html_snippets1(_snippet_):\n",
    "    root1 = fromstring(_snippet_) \n",
    "    title = [x.text for x in root1.xpath('//div[@class=\"inner_link_article_title\"]')]\n",
    "    create_time = [x.text for x in root1.xpath('//div[@class=\"inner_link_article_date\"]')]\n",
    "    link = [x for x in root1.xpath('//a/@href')]\n",
    "    _df1_ = pd.DataFrame({\"title\":title, \"create_time\": create_time, \"link\":link})\n",
    "    return(_df1_)\n",
    "    \n",
    "l_df1 = []\n",
    "for p in pages:\n",
    "    _df1_ = parse_html_snippets(df1.loc[p,\"html_snippets\"])\n",
    "    print (len(_df1_), end=\",\")\n",
    "    l_df1.append(_df1_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8个超好用的Python内置函数，提升效率必备！</td>\n",
       "      <td>2020-04-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>粉丝专属福利，四折优惠码限时免费领！</td>\n",
       "      <td>2020-04-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>你有多久没读过书了？</td>\n",
       "      <td>2020-04-10</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>今天网站都变成灰色了，这其中是怎么实现的？</td>\n",
       "      <td>2020-04-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>是技术也是艺术 使用geopandas玩转地图可视化</td>\n",
       "      <td>2020-03-06</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Python教你解决微信防撤回（必备技能）</td>\n",
       "      <td>2020-01-03</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>8个超好用的Python内置函数，提升效率必备！</td>\n",
       "      <td>2020-04-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>粉丝专属福利，四折优惠码限时免费领！</td>\n",
       "      <td>2020-04-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>你有多久没读过书了？</td>\n",
       "      <td>2020-04-10</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>今天网站都变成灰色了，这其中是怎么实现的？</td>\n",
       "      <td>2020-04-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>是技术也是艺术 使用geopandas玩转地图可视化</td>\n",
       "      <td>2020-03-06</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Python教你解决微信防撤回（必备技能）</td>\n",
       "      <td>2020-01-03</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>直男不懂口红色号？没关系！——数据分析告诉你节日送什么色号口红</td>\n",
       "      <td>2019-06-19</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>第一次给大家送福利，想想都有点小激动！</td>\n",
       "      <td>2019-06-18</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>BigData | 一文带你搞清楚\"数据倾斜\"</td>\n",
       "      <td>2019-06-17</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>BigData |述说Apache Spark</td>\n",
       "      <td>2019-06-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>BigData | 大数据处理基本功（上）</td>\n",
       "      <td>2019-06-13</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>BigData | 大数据处理基本功（下）</td>\n",
       "      <td>2019-06-13</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>GitHub 发布了官方 App，还打算冰封你的代码一千年</td>\n",
       "      <td>2019-11-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>用Power BI了解站在猪年风口上的猪有多金贵</td>\n",
       "      <td>2019-11-11</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>这次我们不需要砍价，盖楼，​直接使用优惠码即可享受优惠。​</td>\n",
       "      <td>2019-11-08</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              title create_time  \\\n",
       "0          8个超好用的Python内置函数，提升效率必备！  2020-04-15   \n",
       "1                粉丝专属福利，四折优惠码限时免费领！  2020-04-15   \n",
       "2                        你有多久没读过书了？  2020-04-10   \n",
       "3             今天网站都变成灰色了，这其中是怎么实现的？  2020-04-04   \n",
       "4        是技术也是艺术 使用geopandas玩转地图可视化  2020-03-06   \n",
       "5             Python教你解决微信防撤回（必备技能）  2020-01-03   \n",
       "6          8个超好用的Python内置函数，提升效率必备！  2020-04-15   \n",
       "7                粉丝专属福利，四折优惠码限时免费领！  2020-04-15   \n",
       "8                        你有多久没读过书了？  2020-04-10   \n",
       "9             今天网站都变成灰色了，这其中是怎么实现的？  2020-04-04   \n",
       "10       是技术也是艺术 使用geopandas玩转地图可视化  2020-03-06   \n",
       "11            Python教你解决微信防撤回（必备技能）  2020-01-03   \n",
       "12  直男不懂口红色号？没关系！——数据分析告诉你节日送什么色号口红  2019-06-19   \n",
       "13              第一次给大家送福利，想想都有点小激动！  2019-06-18   \n",
       "14          BigData | 一文带你搞清楚\"数据倾斜\"  2019-06-17   \n",
       "15          BigData |述说Apache Spark  2019-06-15   \n",
       "16            BigData | 大数据处理基本功（上）  2019-06-13   \n",
       "17            BigData | 大数据处理基本功（下）  2019-06-13   \n",
       "18    GitHub 发布了官方 App，还打算冰封你的代码一千年  2019-11-15   \n",
       "19         用Power BI了解站在猪年风口上的猪有多金贵  2019-11-11   \n",
       "20    这次我们不需要砍价，盖楼，​直接使用优惠码即可享受优惠。​  2019-11-08   \n",
       "\n",
       "                                                 link  \n",
       "0   http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "1   http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "2   http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "3   http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "4   http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "5   http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "6   http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "7   http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "8   http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "9   http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "10  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "11  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "12  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "13  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "14  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "15  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "16  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "17  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "18  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "19  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "20  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  "
      ]
     },
     "execution_count": 152,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1_url_out = pd.concat(l_df1).reset_index(drop=True)\n",
    "df1_url_out.loc[0:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>value</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>粉丝专属福利，四折优惠码限时免费领！</td>\n",
       "      <td>2020-04-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>你有多久没读过书了？</td>\n",
       "      <td>2020-04-10</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>今天网站都变成灰色了，这其中是怎么实现的？</td>\n",
       "      <td>2020-04-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>粉丝专属福利，四折优惠码限时免费领！</td>\n",
       "      <td>2020-04-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>你有多久没读过书了？</td>\n",
       "      <td>2020-04-10</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>168</th>\n",
       "      <td>科学计算工具Numpy</td>\n",
       "      <td>2018-07-01</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>172</th>\n",
       "      <td>如何在1年内获得别人3年的工作经验？</td>\n",
       "      <td>2018-06-28</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>173</th>\n",
       "      <td>MySQL进阶</td>\n",
       "      <td>2018-06-26</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>174</th>\n",
       "      <td>数据库 - MySQL的简单使用</td>\n",
       "      <td>2018-06-25</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>175</th>\n",
       "      <td>微信好友全头像的拼接</td>\n",
       "      <td>2018-06-22</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>110 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                       title create_time  \\\n",
       "value                                      \n",
       "1         粉丝专属福利，四折优惠码限时免费领！  2020-04-15   \n",
       "2                 你有多久没读过书了？  2020-04-10   \n",
       "3      今天网站都变成灰色了，这其中是怎么实现的？  2020-04-04   \n",
       "7         粉丝专属福利，四折优惠码限时免费领！  2020-04-15   \n",
       "8                 你有多久没读过书了？  2020-04-10   \n",
       "...                      ...         ...   \n",
       "168              科学计算工具Numpy  2018-07-01   \n",
       "172       如何在1年内获得别人3年的工作经验？  2018-06-28   \n",
       "173                  MySQL进阶  2018-06-26   \n",
       "174         数据库 - MySQL的简单使用  2018-06-25   \n",
       "175               微信好友全头像的拼接  2018-06-22   \n",
       "\n",
       "                                                    link  \n",
       "value                                                     \n",
       "1      http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "2      http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "3      http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "7      http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "8      http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "...                                                  ...  \n",
       "168    http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "172    http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "173    http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "174    http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "175    http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "\n",
       "[110 rows x 3 columns]"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# tagging 标记\n",
    "tagging_list = [\"\",'数据科学','Python','数据分析','可视化','机器学习','深度学习','自然语言处理'\\\n",
    " '科研入门知识','选题','分析文献','采集数据','阅读论文','写作稿件','答辩'\\\n",
    "'效率','笔记','写作','幻灯制作','视频剪辑'\\\n",
    "'杂项','观察','思考','记录','视频', '书籍' ,'就业' '爬虫'] #overwritable\n",
    "\n",
    "v_v_list1 = []\n",
    "#.melt()取消将DataFrame从宽格式转换为长格式。\n",
    "for tag in tagging_list:\n",
    "    index_list1 = df1_url_out[ df1_url_out.title.str.contains(tag) ].index.tolist()\n",
    "    v_v_pairs1 = pd.DataFrame({tag:index_list1}).melt().set_index(\"value\")\n",
    "    v_v_list1.append(v_v_pairs1)\n",
    "\n",
    "df1_cat = v_v_list1[0]\n",
    "for d in v_v_list1:\n",
    "    df1_cat.update(d)\n",
    "    \n",
    "# 尚未标记内容\n",
    "df1_url_out.loc [ df1_cat.query('variable==\"\"').index ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>8个超好用的Python内置函数，提升效率必备！</td>\n",
       "      <td>2020-04-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>粉丝专属福利，四折优惠码限时免费领！</td>\n",
       "      <td>2020-04-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>你有多久没读过书了？</td>\n",
       "      <td>2020-04-10</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>今天网站都变成灰色了，这其中是怎么实现的？</td>\n",
       "      <td>2020-04-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>是技术也是艺术 使用geopandas玩转地图可视化</td>\n",
       "      <td>2020-03-06</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Python教你解决微信防撤回（必备技能）</td>\n",
       "      <td>2020-01-03</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>直男不懂口红色号？没关系！——数据分析告诉你节日送什么色号口红</td>\n",
       "      <td>2019-06-19</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66</th>\n",
       "      <td>第一次给大家送福利，想想都有点小激动！</td>\n",
       "      <td>2019-06-18</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>BigData | 一文带你搞清楚\"数据倾斜\"</td>\n",
       "      <td>2019-06-17</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>BigData |述说Apache Spark</td>\n",
       "      <td>2019-06-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69</th>\n",
       "      <td>BigData | 大数据处理基本功（上）</td>\n",
       "      <td>2019-06-13</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>BigData | 大数据处理基本功（下）</td>\n",
       "      <td>2019-06-13</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              title create_time  \\\n",
       "6          8个超好用的Python内置函数，提升效率必备！  2020-04-15   \n",
       "7                粉丝专属福利，四折优惠码限时免费领！  2020-04-15   \n",
       "8                        你有多久没读过书了？  2020-04-10   \n",
       "9             今天网站都变成灰色了，这其中是怎么实现的？  2020-04-04   \n",
       "10       是技术也是艺术 使用geopandas玩转地图可视化  2020-03-06   \n",
       "11            Python教你解决微信防撤回（必备技能）  2020-01-03   \n",
       "65  直男不懂口红色号？没关系！——数据分析告诉你节日送什么色号口红  2019-06-19   \n",
       "66              第一次给大家送福利，想想都有点小激动！  2019-06-18   \n",
       "67          BigData | 一文带你搞清楚\"数据倾斜\"  2019-06-17   \n",
       "68          BigData |述说Apache Spark  2019-06-15   \n",
       "69            BigData | 大数据处理基本功（上）  2019-06-13   \n",
       "70            BigData | 大数据处理基本功（下）  2019-06-13   \n",
       "\n",
       "                                                 link  \n",
       "6   http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "7   http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "8   http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "9   http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "10  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "11  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "65  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "66  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "67  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "68  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "69  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "70  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  "
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1_url_out[df1_url_out.duplicated()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8个超好用的Python内置函数，提升效率必备！</td>\n",
       "      <td>2020-04-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>粉丝专属福利，四折优惠码限时免费领！</td>\n",
       "      <td>2020-04-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>你有多久没读过书了？</td>\n",
       "      <td>2020-04-10</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>今天网站都变成灰色了，这其中是怎么实现的？</td>\n",
       "      <td>2020-04-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>是技术也是艺术 使用geopandas玩转地图可视化</td>\n",
       "      <td>2020-03-06</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>171</th>\n",
       "      <td>Python线程的使用</td>\n",
       "      <td>2018-06-28</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>172</th>\n",
       "      <td>如何在1年内获得别人3年的工作经验？</td>\n",
       "      <td>2018-06-28</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>173</th>\n",
       "      <td>MySQL进阶</td>\n",
       "      <td>2018-06-26</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>174</th>\n",
       "      <td>数据库 - MySQL的简单使用</td>\n",
       "      <td>2018-06-25</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>175</th>\n",
       "      <td>微信好友全头像的拼接</td>\n",
       "      <td>2018-06-22</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>164 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                          title create_time  \\\n",
       "0      8个超好用的Python内置函数，提升效率必备！  2020-04-15   \n",
       "1            粉丝专属福利，四折优惠码限时免费领！  2020-04-15   \n",
       "2                    你有多久没读过书了？  2020-04-10   \n",
       "3         今天网站都变成灰色了，这其中是怎么实现的？  2020-04-04   \n",
       "4    是技术也是艺术 使用geopandas玩转地图可视化  2020-03-06   \n",
       "..                          ...         ...   \n",
       "171                 Python线程的使用  2018-06-28   \n",
       "172          如何在1年内获得别人3年的工作经验？  2018-06-28   \n",
       "173                     MySQL进阶  2018-06-26   \n",
       "174            数据库 - MySQL的简单使用  2018-06-25   \n",
       "175                  微信好友全头像的拼接  2018-06-22   \n",
       "\n",
       "                                                  link  \n",
       "0    http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "1    http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "2    http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "3    http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "4    http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "..                                                 ...  \n",
       "171  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "172  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "173  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "174  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "175  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...  \n",
       "\n",
       "[164 rows x 3 columns]"
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1_url_out[~df1_url_out.duplicated()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "      <th>variable</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8个超好用的Python内置函数，提升效率必备！</td>\n",
       "      <td>2020-04-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "      <td>Python</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>粉丝专属福利，四折优惠码限时免费领！</td>\n",
       "      <td>2020-04-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>你有多久没读过书了？</td>\n",
       "      <td>2020-04-10</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>今天网站都变成灰色了，这其中是怎么实现的？</td>\n",
       "      <td>2020-04-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>是技术也是艺术 使用geopandas玩转地图可视化</td>\n",
       "      <td>2020-03-06</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "      <td>可视化</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>171</th>\n",
       "      <td>Python线程的使用</td>\n",
       "      <td>2018-06-28</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "      <td>Python</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>172</th>\n",
       "      <td>如何在1年内获得别人3年的工作经验？</td>\n",
       "      <td>2018-06-28</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>173</th>\n",
       "      <td>MySQL进阶</td>\n",
       "      <td>2018-06-26</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>174</th>\n",
       "      <td>数据库 - MySQL的简单使用</td>\n",
       "      <td>2018-06-25</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>175</th>\n",
       "      <td>微信好友全头像的拼接</td>\n",
       "      <td>2018-06-22</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>176 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                          title create_time  \\\n",
       "0      8个超好用的Python内置函数，提升效率必备！  2020-04-15   \n",
       "1            粉丝专属福利，四折优惠码限时免费领！  2020-04-15   \n",
       "2                    你有多久没读过书了？  2020-04-10   \n",
       "3         今天网站都变成灰色了，这其中是怎么实现的？  2020-04-04   \n",
       "4    是技术也是艺术 使用geopandas玩转地图可视化  2020-03-06   \n",
       "..                          ...         ...   \n",
       "171                 Python线程的使用  2018-06-28   \n",
       "172          如何在1年内获得别人3年的工作经验？  2018-06-28   \n",
       "173                     MySQL进阶  2018-06-26   \n",
       "174            数据库 - MySQL的简单使用  2018-06-25   \n",
       "175                  微信好友全头像的拼接  2018-06-22   \n",
       "\n",
       "                                                  link variable  \n",
       "0    http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...   Python  \n",
       "1    http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...     无法分类  \n",
       "2    http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...     无法分类  \n",
       "3    http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...     无法分类  \n",
       "4    http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...      可视化  \n",
       "..                                                 ...      ...  \n",
       "171  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...   Python  \n",
       "172  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...     无法分类  \n",
       "173  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...     无法分类  \n",
       "174  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...     无法分类  \n",
       "175  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...     无法分类  \n",
       "\n",
       "[176 rows x 4 columns]"
      ]
     },
     "execution_count": 156,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_o1 = df1_url_out.join(df1_cat).replace(\"\", np.nan).fillna(\"无法分类\")\n",
    "df_o1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>variable</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>无法分类</th>\n",
       "      <td>110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Python</th>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>数据分析</th>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>可视化</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>视频</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>记录</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>观察</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          title\n",
       "variable       \n",
       "无法分类        110\n",
       "Python       47\n",
       "数据分析          9\n",
       "可视化           5\n",
       "视频            2\n",
       "记录            2\n",
       "观察            1"
      ]
     },
     "execution_count": 157,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1_stats = df_o1.groupby(by=\"variable\").agg({\"title\":\"count\"}).sort_values(by=\"title\", ascending=False)\n",
    "df1_stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 输出"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "      <th>variable</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>大一就开始准备考研，合适吗？</td>\n",
       "      <td>2020-05-13</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>如何用人工智能免费帮你改英文？</td>\n",
       "      <td>2020-05-05</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>如何高效做文献回顾？（工具篇）</td>\n",
       "      <td>2020-04-28</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>卡片式笔记工具 Roam Research 介绍</td>\n",
       "      <td>2020-04-20</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>笔记</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>如何交互可视化你的卡片式笔记网络？</td>\n",
       "      <td>2020-04-12</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>笔记</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200</th>\n",
       "      <td>我的MOOC之路（三）——爱恨交加机器学习</td>\n",
       "      <td>2016-07-06</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>机器学习</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>201</th>\n",
       "      <td>Ulysses一键发送文章到Day One</td>\n",
       "      <td>2016-07-05</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>202</th>\n",
       "      <td>我的MOOC之路（二）——七周学会做搜索引擎</td>\n",
       "      <td>2016-07-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>203</th>\n",
       "      <td>微信公众号Markdown Here排版的缺失环节</td>\n",
       "      <td>2016-07-03</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>204</th>\n",
       "      <td>我的MOOC之路（一）—— 故事从这里开始</td>\n",
       "      <td>2016-07-02</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>205 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                         title create_time  \\\n",
       "0               大一就开始准备考研，合适吗？  2020-05-13   \n",
       "1              如何用人工智能免费帮你改英文？  2020-05-05   \n",
       "2              如何高效做文献回顾？（工具篇）  2020-04-28   \n",
       "3     卡片式笔记工具 Roam Research 介绍  2020-04-20   \n",
       "4            如何交互可视化你的卡片式笔记网络？  2020-04-12   \n",
       "..                         ...         ...   \n",
       "200      我的MOOC之路（三）——爱恨交加机器学习  2016-07-06   \n",
       "201      Ulysses一键发送文章到Day One  2016-07-05   \n",
       "202     我的MOOC之路（二）——七周学会做搜索引擎  2016-07-04   \n",
       "203  微信公众号Markdown Here排版的缺失环节  2016-07-03   \n",
       "204      我的MOOC之路（一）—— 故事从这里开始  2016-07-02   \n",
       "\n",
       "                                                  link variable  \n",
       "0    http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     无法分类  \n",
       "1    http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     无法分类  \n",
       "2    http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     无法分类  \n",
       "3    http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...       笔记  \n",
       "4    http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...       笔记  \n",
       "..                                                 ...      ...  \n",
       "200  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     机器学习  \n",
       "201  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     无法分类  \n",
       "202  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     无法分类  \n",
       "203  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     无法分类  \n",
       "204  http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     无法分类  \n",
       "\n",
       "[205 rows x 4 columns]"
      ]
     },
     "execution_count": 158,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_o"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "      <th>variable</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8个超好用的Python内置函数，提升效率必备！</td>\n",
       "      <td>2020-04-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "      <td>Python</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>粉丝专属福利，四折优惠码限时免费领！</td>\n",
       "      <td>2020-04-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>你有多久没读过书了？</td>\n",
       "      <td>2020-04-10</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>今天网站都变成灰色了，这其中是怎么实现的？</td>\n",
       "      <td>2020-04-04</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>是技术也是艺术 使用geopandas玩转地图可视化</td>\n",
       "      <td>2020-03-06</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "      <td>可视化</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>171</th>\n",
       "      <td>Python线程的使用</td>\n",
       "      <td>2018-06-28</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "      <td>Python</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>172</th>\n",
       "      <td>如何在1年内获得别人3年的工作经验？</td>\n",
       "      <td>2018-06-28</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>173</th>\n",
       "      <td>MySQL进阶</td>\n",
       "      <td>2018-06-26</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>174</th>\n",
       "      <td>数据库 - MySQL的简单使用</td>\n",
       "      <td>2018-06-25</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>175</th>\n",
       "      <td>微信好友全头像的拼接</td>\n",
       "      <td>2018-06-22</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>176 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                          title create_time  \\\n",
       "0      8个超好用的Python内置函数，提升效率必备！  2020-04-15   \n",
       "1            粉丝专属福利，四折优惠码限时免费领！  2020-04-15   \n",
       "2                    你有多久没读过书了？  2020-04-10   \n",
       "3         今天网站都变成灰色了，这其中是怎么实现的？  2020-04-04   \n",
       "4    是技术也是艺术 使用geopandas玩转地图可视化  2020-03-06   \n",
       "..                          ...         ...   \n",
       "171                 Python线程的使用  2018-06-28   \n",
       "172          如何在1年内获得别人3年的工作经验？  2018-06-28   \n",
       "173                     MySQL进阶  2018-06-26   \n",
       "174            数据库 - MySQL的简单使用  2018-06-25   \n",
       "175                  微信好友全头像的拼接  2018-06-22   \n",
       "\n",
       "                                                  link variable  \n",
       "0    http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...   Python  \n",
       "1    http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...     无法分类  \n",
       "2    http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...     无法分类  \n",
       "3    http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...     无法分类  \n",
       "4    http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...      可视化  \n",
       "..                                                 ...      ...  \n",
       "171  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...   Python  \n",
       "172  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...     无法分类  \n",
       "173  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...     无法分类  \n",
       "174  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...     无法分类  \n",
       "175  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...     无法分类  \n",
       "\n",
       "[176 rows x 4 columns]"
      ]
     },
     "execution_count": 159,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_o1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "      <th>variable</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>大一就开始准备考研，合适吗？</td>\n",
       "      <td>2020-05-13</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>如何用人工智能免费帮你改英文？</td>\n",
       "      <td>2020-05-05</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>如何高效做文献回顾？（工具篇）</td>\n",
       "      <td>2020-04-28</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>卡片式笔记工具 Roam Research 介绍</td>\n",
       "      <td>2020-04-20</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>笔记</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>如何交互可视化你的卡片式笔记网络？</td>\n",
       "      <td>2020-04-12</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...</td>\n",
       "      <td>笔记</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>376</th>\n",
       "      <td>Python线程的使用</td>\n",
       "      <td>2018-06-28</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "      <td>Python</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>377</th>\n",
       "      <td>如何在1年内获得别人3年的工作经验？</td>\n",
       "      <td>2018-06-28</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>378</th>\n",
       "      <td>MySQL进阶</td>\n",
       "      <td>2018-06-26</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>379</th>\n",
       "      <td>数据库 - MySQL的简单使用</td>\n",
       "      <td>2018-06-25</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>380</th>\n",
       "      <td>微信好友全头像的拼接</td>\n",
       "      <td>2018-06-22</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>381 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                        title create_time  \\\n",
       "0              大一就开始准备考研，合适吗？  2020-05-13   \n",
       "1             如何用人工智能免费帮你改英文？  2020-05-05   \n",
       "2             如何高效做文献回顾？（工具篇）  2020-04-28   \n",
       "3    卡片式笔记工具 Roam Research 介绍  2020-04-20   \n",
       "4           如何交互可视化你的卡片式笔记网络？  2020-04-12   \n",
       "..                        ...         ...   \n",
       "376               Python线程的使用  2018-06-28   \n",
       "377        如何在1年内获得别人3年的工作经验？  2018-06-28   \n",
       "378                   MySQL进阶  2018-06-26   \n",
       "379          数据库 - MySQL的简单使用  2018-06-25   \n",
       "380                微信好友全头像的拼接  2018-06-22   \n",
       "\n",
       "                                                  link variable  \n",
       "0    http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     无法分类  \n",
       "1    http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     无法分类  \n",
       "2    http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...     无法分类  \n",
       "3    http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...       笔记  \n",
       "4    http://mp.weixin.qq.com/s?__biz=MzIyODI1MzYyNA...       笔记  \n",
       "..                                                 ...      ...  \n",
       "376  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...   Python  \n",
       "377  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...     无法分类  \n",
       "378  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...     无法分类  \n",
       "379  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...     无法分类  \n",
       "380  http://mp.weixin.qq.com/s?__biz=MzI5ODMyMzgxOQ...     无法分类  \n",
       "\n",
       "[381 rows x 4 columns]"
      ]
     },
     "execution_count": 161,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_o2=df_o2=pd.concat([df_o,df_o1])\n",
    "df_o2.reset_index(drop=True, inplace=True)\n",
    "df_o2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_stats2=pd.concat([df_stats,df1_stats,])\n",
    "df_stats2.reset_index(drop=True, inplace=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>variable</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>无法分类</th>\n",
       "      <td>235</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Python</th>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>数据分析</th>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>机器学习</th>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>视频</th>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>可视化</th>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>数据科学</th>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>写作</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>笔记</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>深度学习</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>记录</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>思考</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>观察</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>选题</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          title\n",
       "variable       \n",
       "无法分类        235\n",
       "Python       85\n",
       "数据分析          9\n",
       "机器学习          9\n",
       "视频            9\n",
       "可视化           7\n",
       "数据科学          7\n",
       "写作            5\n",
       "笔记            5\n",
       "深度学习          4\n",
       "记录            3\n",
       "思考            1\n",
       "观察            1\n",
       "选题            1"
      ]
     },
     "execution_count": 186,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_stats2 = df_o2.groupby(by=\"variable\").agg({\"title\":\"count\"}).sort_values(by=\"title\", ascending=False)\n",
    "df_stats2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nickname</th>\n",
       "      <th>wechat</th>\n",
       "      <th>img</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>芝兰玉树书画载道</td>\n",
       "      <td>微信号：未设置</td>\n",
       "      <td>http://mmbiz.qpic.cn/mmbiz_png/LzWKYLw2Q0OUpaW...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>芝兰玉树书画载道</td>\n",
       "      <td>微信号：未设置</td>\n",
       "      <td>http://mmbiz.qpic.cn/mmbiz_png/LzWKYLw2Q0OUpaW...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>芝兰玉树书画载道</td>\n",
       "      <td>微信号：未设置</td>\n",
       "      <td>http://mmbiz.qpic.cn/mmbiz_png/LzWKYLw2Q0OUpaW...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>芝兰玉树书画载道</td>\n",
       "      <td>微信号：未设置</td>\n",
       "      <td>http://mmbiz.qpic.cn/mmbiz_png/LzWKYLw2Q0OUpaW...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   nickname   wechat                                                img\n",
       "0  芝兰玉树书画载道  微信号：未设置  http://mmbiz.qpic.cn/mmbiz_png/LzWKYLw2Q0OUpaW...\n",
       "1  芝兰玉树书画载道  微信号：未设置  http://mmbiz.qpic.cn/mmbiz_png/LzWKYLw2Q0OUpaW...\n",
       "2  芝兰玉树书画载道  微信号：未设置  http://mmbiz.qpic.cn/mmbiz_png/LzWKYLw2Q0OUpaW...\n",
       "3  芝兰玉树书画载道  微信号：未设置  http://mmbiz.qpic.cn/mmbiz_png/LzWKYLw2Q0OUpaW..."
      ]
     },
     "execution_count": 190,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_account2=pd.concat([df_account,df_account1])\n",
    "df_account2.reset_index(drop=True, inplace=True)\n",
    "df_account2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_account2.columns.name = \"rel_accounts\"\n",
    "df_o2.columns.name = \"url_cat\"\n",
    "df_stats2.columns.name = \"stats\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {},
   "outputs": [],
   "source": [
    "with pd.ExcelWriter(fn[\"output\"][\"公众号_xlsx\"].format(公众号=公众号)) as writer:\n",
    "#    workbook  = writer.book\n",
    "\n",
    "    for _df_ in [df_account2, df_o2, df_stats2]:\n",
    "        _df_.to_excel(writer, sheet_name = _df_.columns.name)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
